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Foreword 



The last decade of computer vision can be characterized by the development 
of tools for geometric reasoning. Algebraic projective geometry, with its mul- 
tilinear relations and its embedding into Grassmann-Cayley algebra, became 
the basic representation of multiple view geometry, on the one hand lead- 
ing to a deep insight into the algebraic structure of the geometric relations 
and on the other hand leading to efficient and versatile algorithms for the 
calibration and orientation of cameras, for image sequence analysis as well 
as for 3D-object reconstruction. Prior to the use of projective geometry in 
computer vision, some kinds of features or information had to be extracted 
from the images. The inherent uncertainty of these automatically extracted 
image features, in a first instance image points and image line segments, re- 
quired tracing the effects up to the inherently uncertain result. However, in 
spite of the awareness of the problem and the attempts to integrate uncer- 
tainty into geometric computations, especially joint final maximum-likelihood 
estimation of camera and object parameters, geometric reasoning under un- 
certainty rarely was seen as a topic on its own. 

Reasons for the lack of integration of uncertainty may have been the redun- 
dancy in the representation of Euclidean entities using homogeneous coordi- 
nates leading to singularities in the statistical distributions, the non-linearity 
of the relations requiring approximations, which give biased results, and the 
dependency of both the numerical accuracy and the statistical properties 
on the coordinate systems chosen; another reason may have been the lower 
reward for performing good engineering. 

The author provides a transparent integration of algebraic projective geom- 
etry and spatial reasoning under uncertainty with applications in computer 
vision. Examples of those applications are: grouping of spatial features, re- 
construction of polyhedral objects, or calibration of projective cameras. He 
clearly cuts the boundary between rigorous but infeasible and approximate 
but practicable statistical tools for three-dimensional computations. In addi- 
tion to the theoretical foundations of the proposed integration and the clear 
rules for performing geometric reasoning under uncertainty, he provides a 
collection of detailed algorithms that are implemented using now publicly 
available software. 
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Foreword 



The book will be of value to all scientists interested in algebraic projective 
geometry for image analysis, in statistical representations of two- and three- 
dimensional objects and transformations, and in generic tools for testing and 
estimating within the context of geometric multiple- view analysis. 



Bonn, February 2004 



Wolfgang Forstner 



Preface 



I wish to acknowledge the following people who directly or indirectly con- 
tributed to this work. First, I would like to thank my family, especially my 
parents, who have given me sufficient freedom to pursue my own interests 
and at the same time have shown me unconditional love. 

I was very fortunate to have met wonderful people during my time at the 
University of Bonn. I refer in particular to Oliver who contributed to my life 
in numerous ways. 

I also thank all the people at the University and the Institute of Photogram- 
metry in Bonn for a wonderful time I had there both as a student assistant 
and as a research assistant. In particular I wish to mention Kirsten Wolff, An- 
dre Braunmandl, Heidi Hollander, Felicitas Lang, Wolfgang Schickler, Hardo 
Muller, Boudewijn Moonen, Thomas H. Kolbe, Petko and Annett Faber, 
Mirko Appel and many others. I also want to thank Thomas Labe for coor- 
dinating the Java implementation of the algorithms described in this work. 

I am deeply grateful that I had the opportunity to work with my advisor. 
Prof. Dr. Wolfgang Forstner. He was not only the initiator of this work and 
provided continuous support throughout the duration of the research, but 
more importantly he was a Doktorvater in the truest sense of the word. I 
extend my gratitude to Prof. Dr. Helfrich for his willingness to serve as a 
second reviewer for this work. 

Lastly, I wish to express my thanks to Anette for her support during the last 
15 months. I am still wondering why we did not meet earlier in life. 



Bonn, October 2002 



Stephan Heuel 



Contents 



1 Introduction 1 

1 . 1 Motivation 1 

1.2 Objective of the Thesis 3 

1.3 Reconstructions of Polyhedral Objects 4 

1.3.1 Object, Sensor, and Image Models 4 

1.3.2 Representation of Polygons and Polyhedra 6 

1.3.3 Automated Reconstruction 7 

1.4 Geometric Reasoning 8 

1.4.1 Reasoning and Object Reconstruction 9 

1.4.2 Reasoning within Geometry 10 

1.4.3 Uncertain Reasoning 11 

1.5 Previous Work 11 

1.5.1 Projective Geometry 11 

1.5.2 Statistical Geometry and Reasoning 13 

1.5.3 Polyhedral Object Reconstruction 15 

1.6 Overview of Thesis 17 

2 Representation ofGeometric Entities and Transformations . 19 

2.1 Projective Geometry 19 

2.1.1 Projective Space and Homogeneous Goordinates 20 

2.1.2 Hyperplanes 23 

2.2 Representation of Geometric Entities 24 

2.2.1 Points and Lines in 2D 25 

2.2.2 Points and Planes in 3D 28 

2.2.3 Lines in 3D 30 

2.2.4 Pliicker Goordinates 32 

2.3 Basic Geometric Transformations 34 

2.3.1 Homography 34 

2.3.2 Projective Gamera 35 

2.3.3 Fundamental Matrix 36 



X 



Contents 



2.4 Conditioning of Homogeneous Entities 38 

2.5 Duality Principle 40 

2.5.1 Dual Entities 41 

2.5.2 Dual of Point Transformations 44 

3 Geometric Reasoning Using Projective Geometry 47 

3.1 Unique Constructions of Entities 48 

3.1.1 Join and Intersection 49 

3.1.2 Transformation of Points, Lines, and Planes 56 

3.1.3 Fundamental Matrix 59 

3.1.4 Inverse Projective Camera with Respect to a Plane ... 59 

3.2 Construction Matrices and Their Interpretation 60 

3.2.1 Canonical Entities in Construction Matrices 61 

3.2.2 Reduction of Construction Matrices 67 

3.2.3 Nullspaces of Construction Matrices 69 

3.3 Relations between Entities 70 

3.3.1 Projective Relations 71 

3.3.2 Afhne and Similarity Relations 77 

3.3.3 Distance Relations 79 

3.3.4 Checking Geometric Relations 79 

3.4 General Construction of Entities 80 

3.4.1 Relations as Constraints 81 

3.4.2 Minimizing Algebraic Distance 83 

3.4.3 Enforcing Pliicker Constraint 84 

3.5 Estimating Projective Transformations 85 

3.5.1 Collinearity Equations 85 

3.5.2 Coplanarity Equations 88 

3.5.3 Simultaneous DLT with Points and Lines 89 

3.5.4 Simultaneous DLT Algorithms for Homographies 90 

3.5.5 Estimating Constrained Transformations 92 

3.5.6 Conditioning of Entities for Minimization 93 

3.5.7 Generic Construction Algorithm 95 

4 Statistical Geometric Reasoning 97 

4.1 Representation of Uncertain Geometric Entities 98 

4.1.1 General Representation 98 

4.1.2 Approximate Representation 102 

4.2 Transformation of Uncertain Homogeneous Vectors 104 

4.2.1 First Order Error Propagation 104 



Contents 



XI 



4.2.2 Transfer to Homogeneous Coordinates 105 

4.2.3 Normalization to Euclidean Coordinates 109 

4.2.4 Normalization to Spherical Coordinates 110 

4.2.5 Changing Nullspaces Using Orthogonal Projections . . . 110 

4.2.6 Construction Ill 

4.3 Errors in Approximated Uncertainty Representation 113 

4.3.1 Second Moments and Gaussian Assumption 113 

4.3.2 Bias in Scalar Multiplication 114 

4.3.3 Bias in Bilinear Constructions 117 

4.3.4 Bias in Normalization 120 

4.4 Construction of Entities 122 

4.4.1 A Statistical Approach to Join and Intersection 122 

4.4.2 Validation of the Statistical Approach 125 

4.4.3 Construction Using Geometric Transformations 128 

4.5 Testing Geometric Relations 129 

4.5.1 Hypothesis Testing 129 

4.5.2 Properties of the Approximated Test- Value T 132 

4.5.3 A Statistical Algorithm for Testing Geometric 

Relations 135 

4.5.4 Validation of Hypothesis Tests 135 

4.5.5 Further Improvements 137 

4.6 Optimal Geometric Estimation 139 

4.6.1 Statistical Model 139 

4.6.2 Iterative Estimation 143 

4.6.3 Generic Algorithm for Optimal Geometric Estimation . 144 

4.6.4 Example: Estimating Entities of a Cube 146 

4.7 SUGR: a Library for Statistical Uncertain Geometric 

Reasoning 147 

Polyhedral Object Reconstruction 149 

5.1 Principle Workflow 150 

5.1.1 Feature Extraction 150 

5.1.2 Acquiring the Camera Parameters 151 

5.2 Enhancing User Assisted Reconstruction Systems 153 

5.2.1 Existing User Assisted Systems 153 

5.2.2 User Assisted Constructions of Building Edges 155 

5.2.3 Grouping 3D Line Segments to Surface Patches 159 

5.3 Automated Reconstruction 161 

5.3.1 Matching of Corresponding Line Segments 161 



XII 



Contents 



5.3.2 Examples 164 

5.3.3 Discussion 167 

5.3.4 Effect of Topological Selection to Matching 169 

5.4 Combining Interaction and Automation 170 

5.5 Summary 172 

6 Conclusions 173 

6.1 Summary 173 

6.2 Contribution of Thesis 174 

6.3 Outlook 176 

A Notation 179 

B Linear Algebra 183 

B.l Ranks and Nullspaces 183 

B.2 Orthogonal Projections 184 

B. 3 Kronecker Product and vec(-) Operator 184 

C Statistics 187 

C. l Covariance Matrices for 2D Lines 187 

C.1.1 Uncertainty of a 2D Line 187 

C.l. 2 Euclidean Interpretation of Homogeneous 

Covariances 188 

C.2 Gauss Helmert Estimation 190 

C.2.1 General Gauss Helmert Model 191 

C.2. 2 General Gauss Helmert Model with Block Structure . . 194 



References 



197 



List of Tables 



1.1 Different degrees of automation for a reconstruction process . . 8 

1.2 Three different methods of reasoning 9 

2.1 Canonical 2D points and lines 28 

2.2 Canonical 3D points and planes 30 

2.3 Canonical 3D lines 32 

2.4 Homogeneous representation of points, lines and planes 32 

3.1 Unique constructions using join and intersection 56 

3.2 Constructions using a projective camera 61 

3.3 Interpretation of matrices S(-), r(L) r(L), TT(X) and TT(A) . . 66 

3.4 Nullspaces of construction matrices 70 

3.5 Geometric relations between entities 78 

3.6 3D relations as constraints between observed and unknown 

entities 82 

3.7 Calculating redundancy for geometric estimation problems. ... 84 

3.8 Trilinear relations between P and geometric entities 89 

3.9 Trilinear relations between a homography H and geometric 

entities 91 

4.1 Jacobians for Euclidean normalizations No(-) 110 

4.2 Results of 2D Monte Carlo tests 126 

4.3 Results of 3D Monte Carlo tests 127 

4.4 Relations and algebraic expressions for unknown and 

observations 141 



List of Figures 



1.1 Wireframe, surface and volume models 7 

2.1 Different interpretations of the projective line 21 

2.2 Transfer between Euclidean and Homogeneous Coordinates ... 23 

2.3 Euclidean normalization of a 2D point and a 2D line 27 

2.4 Representation of a 3D line 31 

2.5 Pinhole camera model and the three camera planes A, B and 

C 36 

2.6 Epipolar geometry 37 

2.7 Effect of approximate conditioning homogeneous vectors 39 

2.8 Dual tests in 2D 41 

2.9 Visualization of dual entities 41 

3.1 Geometric Reasoning using projective geometry 48 

3.2 Eight unique constructions using join and intersection 49 

3.3 Inverse projective camera 60 

3.4 Interpretation of construction matrix S 62 

3.5 Interpretation of construction matrices TT and TT 64 

3.6 Interpretation of construction-T (Pliicker-matrix) 65 

3.7 Checking the incidence of a 3D line with a plane 73 

3.8 Incidence and identity relations for 2D and 3D entities 77 

3.9 Parallelity and Orthogonality relations for 2D and 3D entities . 79 

3.10 Examples of non-unique construction of geometric entities .... 81 

3.11 Effects of conditioning 94 

4.1 Uncertain homogeneous vectors in with different pdfs 99 

4.2 Statistical Geometric Reasoning using homogeneous 

coordinates 101 

4.3 Uncertain homogeneous vectors in P^ with different Gaussian 

pdfs 103 

4.4 Visualization of confidence region of points in 2D and 3D 107 



XVI List of Figures 



4.5 Visualization of confidence region of a 2D line and a 3D plane . 108 

4.6 Visualization of the confidence region of a 3D line 108 

4.7 Approximate Statistical Geometric Reasoning 112 

4.8 Relative bias of bilinear operations 116 

4.9 Effect of conditioning to uncertain homogeneous vectors 119 

4.10 Effect of conditioning and normalizing on the bias of the 

variance 120 

4.11 Plot of relative bias of Euclidean normalization 122 

4.12 Results of 2D Monte Carlo test using varying a 127 

4.13 How to tell whether a point Xi is incident to a line 1 130 

4.14 Testing two projective points in P for identity 133 

4.15 Bias for identity test of 2D points 134 

4.16 Validation of Testing algorithm using Monet Carlo tests 138 

4.17 Results of Monte Carlo test for point estimations 147 

5.1 Principle workflow of polyhedral object extraction 150 

5.2 Overview of the segmentation process of FEX 152 

5.3 Geometric situation for building reconstruction 155 

5.4 Optimal estimation of a 3D line segment 155 

5.5 Interactive reconstruction using points and line segments 156 

5.6 Uncertainty of a line segment 157 

5.7 Example in 2D with one line and two points 159 

5.8 Grouping of 3D line segments to planar patches 160 

5.9 Matching lines in multiple views 162 

5.10 3D Line construction using image lines near the epipolar line . 163 

5.11 Result of matching algorithm on artificial data 166 

5.12 Aerial imagery result of matching algorithm 167 

5.13 Problems when matching 2D line segments 169 

5.14 Interactively reducing search-space for matching 171 

C.l Uncertainty of a 2D line 189 



List of Algorithms 



2.1 Conditioning a set of geometric entities 40 

3.1 Consistent reduction of pairs of construction matrices 69 

3.2 A non-statistical algorithm for checking geometric relations. . . 80 

3.3 Algebraic optimization for entities and transformations 95 

4.1 Join and intersection of uncertain geometric entities 123 

4.2 Monte Carlo test for the construction of a 2D point 126 

4.3 Monte Carlo test for the construction of a 3D point 128 

4.4 Testing geometric relations in 2D and 3D (Version 1) 136 

4.5 Testing geometric relations in 2D and 3D (Version 2) 137 

4.6 Optimally estimating geometric entities 145 

5.1 Grouping 3D line segments 160 

5.2 Matching line segments in multiple views 165 

C.l Euclidean uncertainty of a homogeneous line in 2D 190 

C.2 Iterative Optimization using the GauB-Helmert model 193 

C.3 Iterative Block-Optimization using the GauB-Helmert model . . 195 



1 Introduction 



A better conceptual framework for represent- 
ing and manipulating geometric knowledge can 
lead to implementation designs that provide 
more adequate separation between generic ge- 
ometric principles and specific constraints of 
the application. 

Kapur and Mundy 1989b 



1.1 Motivation 

This thesis is about uncertain geometric reasoning by integrating projective 
geometry and statistics. It is motivated by geometric problems in image anal- 
ysis, in particular by object reconstruction from single or multiple images. 
Extracting information from images is a simple task for humans, that can 
already be accomplished at a very early stage: one year old children are able 
to recognize objects on pictures (Piaget 1954). The visual sense is one 
of the dominant ways of perceiving the surrounding world. At first glance 
it is quite surprising that it is an extremely difficult task for computers to 
achieve a similar ability with respect to image interpretation. Forty years ago 
researchers at the MIT thought that it was at a complexity level of a PhD 
thesis to teach computers to see (Roberts 1963). Within the last decades it 
was realized that there are immense problems that need to be solved in order 
to accomplish this task and the field of Computer Vision emerged from this 
problem. 

Among many reasons, the problems are due to the fact that we are only be- 
ginning to understand how the vision system of human beings work. We may 
explain the process on a biological level: it is known how light is mapped on 
our retina and stimulates perceptive neurons, but it is yet unclear how this 
mapping finally causes our mind to recognize objects. Opposed to low-level 
biological resp. neuronal research, the field of psychology tries to provide 
high-level explanations of the visual system, for example Biederman 1987 
describes a model where the human recognizes objects by breaking it down 
into components, however no biological evidence is known for this model. 

S. Heuel: Uncertain Projective Geometry, LNCS 3008, pp. 1-17, 2004. 
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Another early attempt to describe cognitive aspects are the Gestalt psychol- 
ogists in the 1930s, who described objects and their relations by their saliency 
{Prdgnanz). But again, the psychological models do not provide an instruc- 
tion of how to build an artificial system with comparable visual abilities to 
human beings. For example, it is not clear how to describe objects like a chair 
to computers, such that it can somehow match the description with a digi- 
tal image; there just exist too many different forms of chairs to adequately 
describe it by shape, texture, function etc. To summarize, one may say that 
we are living exemplars of a working visual system, but don’t yet have a 
detailed explanation, why this system works. Though the high expectation of 
building artificial visual systems has not been fulfilled, the Computer Vision 
community still has made significant progress in restricted applications, e.g. 
the guidance of robots or vehicles to name only one of them. 

Despite the previous observations, computers do have advantages over human 
beings when they are used as precise measuring devices instead of interpreters 
of images. Extracting precise measurements from images is one of the central 
traditional tasks in Photogrammetry and has been performed since the inven- 
tion of photography. While it is possible to extract metric information from 
one image, usually two or more images are used in photogrammetric tasks. 
Multiple images allow to apply stereoscopic measuring techniques, which en- 
ables the retrieval of real 3D information about the scene. In the beginning 
of the last century, the first devices for stereoscopic measurements, so-called 
stereo-comparators, were invented and extended to analog plotting devices, 
which are still used for practical applications. In 1957 the concept of an an- 
alytical plotter was patented by Helava: an analytical plotter uses a small 
computer to establish the geometry between the analog images. About 1990 
the first digital photogrammetric workstations have been introduced which 
store the images digitally so that the complete process of measuring can now 
be performed on a computer. 

With the introduction of digital photogrammetric stations, the automation of 
the measurement process becomes more and more important. At this point 
it becomes clear that the future goals and problems of Computer Vision 
and Photogrammetry are largely overlapping and therefore both fields share 
more and more work in recent years. For photogrammetrists, there are two 
tasks that are the main goal of automation: first, the determination of the 
orientation parameters of the cameras at the time when the images were taken 
and second the automation of measuring the desired objects in the images. 
Both tasks are central for Photogrammetry as well as for Computer Vision. In 
this work we cover the measuring or reconstruction of buildings from multiple 
images and the determination of the orientation of single images. We focus 
on the reconstruction task and will see that the algebraic relations that we 
derive simplify the orientation task as a special case of reconstruction. We 
further assume that the buildings can be expressed by a polyhedral model. 



1.2 Objective of the Thesis 
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The problem of image interpretation is now restricted to the reconstruction 
of a specific class of objects, its automation is still a difficult problem: the 
buildings that are observed can be arbitrarily complex, they are surrounded 
by unknown scenery and the images may vary in quality, scale and lighting 
conditions. Due to these and other problems, the automated extraction of 
buildings from images is still an on-going research task, cf. Mayer 1999. On 
the other hand, because of the high demand of up-to-date spatial data, there 
exist a number of commercial user-assisted systems for the reconstruction of 
man-made objects (Baltsavias et al. 2001). Recently, user assisted systems 
adopt more and more automation techniques that have been proven to be 
successful and stable enough to be included in a working environment. 



1.2 Objective of the Thesis 

The objective of the thesis is the derivation of theoretical foundations, design 
and empirical verification of a tool-box for statistical geometric algorithms. 
The practical value of the tool-box is demonstrated by its application to the 
task of object reconstruction from multiple images. 

The main idea of this work is the integration of a consistent geometric repre- 
sentation into a statistical framework: some of the geometric problems that 
are of interest for the reconstruction task are for example (i) the optimal 
estimation of a building corner or edge from a set of image observations and 
(ii) testing geometric relations between independently measured corners or 
edges, such as identity, incidence, parallelity or orthogonality. The need for 
a statistical framework arises from the fact that a solution to these problems 
have to be clearly defined in terms of performance and quality of the results 
(Forstner 1996, Courtney et al. 1997). Statistical concepts have the ad- 
vantages that they are comparable across different systems, can be tested 
against empirical data and help in the evaluation of hypotheses. To simplify 
the statistical description, we use linearized models for estimation and test- 
ing (Koch 1999). Although linear models may only appear sub-optimal at 
first sight, they can be used as long as it can be shown that they are sufficient 
with respect to the data and the task that needs to be solved. For example 
for a large number of geodesic tasks the use of linearized statistical models is 
beyond dispute. The advantage here is the fact that the full power of methods 
from linear algebra are at hand. 

Obviously linear statistical models require a linear representation and manip- 
ulation of geometric data. The techniques known from projective geometry - 
a popular mathematical field in the 19th century - have been proven to be 
very convenient for a lot of geometric tasks in Photogrammetry and Com- 
puter Vision. In this work we first express the operations and formulas from 
projective geometry in terms of linear algebra and secondly include the de- 
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rived linear formulas in statistical models to establish practical and working 
algorithms. 

In the remainder of this chapter we describe the background of this work, 
then briefly review related work and finally give an overview of the rest of 
the thesis. 



1.3 Reconstructions of Polyhedral Objects 

The statistical geometric algorithms that are introduced in this thesis are 
motivated by the task of object reconstruction from multiple images: it will 
be shown that the algorithms enable automatic and user-assisted systems 
to generate the most plausible reconstruction from a given set of images. 
Though this thesis concentrates on the reconstruction task, it is important 
to note that other fields may also make use of this work, such as robotics or 
geometric drawing systems. 

The reconstruction of objects from multiple images is a central task both in 
Computer Vision as well as in Photogrammetry. As reconstruction is embed- 
ded in a complete vision process, we distinguish between the terms extraction, 
detection, reconstruction and recognition of objects: the detection or local- 
ization of objects refer to methods that locate an object within a specific 
image area. Hence, the object itself or the object class is already known. 
In contrast to object detection, object recognition does not necessarily have 
one particular object or class of objects which is about to be detected. The 
term recognition is usually used to describe methods to identify objects. It 
is a problem of finding the right categories. The term reconstruction refers 
to methods for obtaining the shape of an object. In Computer Vision terms, 
this task is called “shape from X”, where X may stand for motion, shad- 
ing, stereo, etc. Finally, extraction refers to the whole process of detection, 
possibly recognition and a subsequent reconstruction of the object. 



1.3.1 Object, Sensor, and Image Models 

Since the early 70’s there has been an on-going discussion on the general ap- 
proach of attacking the problem of object extraction from images, which can 
be summarized as “models versus exemplars” (Ponce et al. 1996, Blake 
et al. 2001). The underlying question is how to represent the objects that 
are viewed in the images. On the one hand there are physically-based mod- 
els, that serve as prototypes for a class of objects. These models are usually 
described with a comparable small set of parameters with known semantics. 
On the other hand the objects may be described only by a set of represen- 
tative examples without using any semantics. This discussion is also often 
characterized by the pairs “object-based vs. view-based” or “parametric vs. 
non-parametric” . 



1.3 Reconstructions of Polyhedral Objects 
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In this work we mainly follow the track of a model-based approach as we are 
specifically interested in exactly measuring the parameters of the unknown 
object. We distinguish between three kinds of models: an object model, a 
sensor model and an image model (Braun et al. 1995 and Fuchs 1998). 

Object Model A quite generic model of a building is a polyhedron which 
consists of straight line segments between two points, and planar faces sur- 
rounded by line segments. This description fits to a large number of actual 
buildings. It has to be noted that a polyhedron does not contain any building- 
specific knowledge. An example for a more specific model is a parameterized 
saddle-roof building which has four parameters, namely length, width, height 
and roof-height. This parameter model is quite specific and can be uniquely 
related to a building. We deliberately choose a polyhedral model though; this 
way the algorithms in this thesis do not contain any building-specific knowl- 
edge, such that they can be applied to other applications. In a separate step, 
outside the scope of this work, the reconstructed polyhedra respectively its 
parts have to be labeled with building-specific meanings such as roof, vertical 
walls, etc., see for example Lang 1999 or Teleki 1997. Besides the geomet- 
ric constraints of the object, we assume that the surfaces of a polyhedron are 
opaque and have a diffuse reflectance property. This assumption ensures the 
actual observability of the polyhedral edges and corners. 

Note that we may also define a scene model, that consists of a number of 
objects and possibly light sources, that influence the appearance of the scene. 

Sensor Model A sensor model describes the process of capturing the image 
and thus characterizes the properties of the sensor important to the applica- 
tion. In Photogrammetry, one usually uses a geometric model of a pin-hole 
camera, resulting in a mapping from 3D to 2D that preserves straight lines. 
As real cameras introduce a small distortion to this model, one may cor- 
rect these errors using a non-linear model such as a radial distortion model, 
cf. Forstner 2001b. One may extend the definition of a sensor model 
to any transformation from one domain to another, including algorithms, 
cf.WEIDNER 1997. 

Image Model Finally the image model defines the geometric, radiometric and 
topological properties of the image. The image model is mainly defined by the 
object and sensor model: since we assume a straight-line preserving mapping, 
the image of a polyhedron is ideally a set of polygons that are connected 
to each other, obtaining a Mondrian-like image. Surfaces of polyhedra are 
mapped to blobs, that are surrounded by polygon edges but not necessarily 
simply connected due to occlusions. For the same reason, the mapping does 
not strictly preserve the neighborhood relationships. 

Errors Introducing models of reality means to obtain an abstraction or gen- 
eralization of the world. The abstraction introduces an error when measuring 
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physical properties, the so-called generalization error or bias. For example, 
the introduction of polyhedral models to describe buildings naturally intro- 
duces errors since at a certain level of detail, no existing building fulfills a 
strict polyhedral model. A second source of errors are the actual errors in the 
measurement or observation error. 

Both kinds of errors have to be taken into account, see for example Winter 
1996 and David et al. 1996. 

1.3.2 Representation of Polygons and Polyhedra 

There exist a number of possible representations for a polyhedron, depending 
on the context in which the term is used. In mathematics, or more precisely 
in geometry, a polyhedron is a volume which is enclosed by plane patches, 
cf. Zeidler 1996. Another valid definition refers to polygons stating that a 
polyhedron consists of a set of polygons that are joined at their edges. 

It is easy to see that a polygon in 2D may be defined similarly exchanging 
the word plane by line, edges by point, etc. This notion can be generalized by 
the term polytope, which Coxeter 1989 defines as the general term of the 
sequence point, line segment, polygon, polyhedron, ...” or more specifically: 

Definition 1 (Polytope) A polytope in an n-dimensional space is a finite 
region enclosed by a finite number of hyperplanes. A polygon is a polytope in 
2 dimensions, a polyhedron a polytope in 3 dimensions. 

The advantage of this definition besides its generality is the fact, that it uses 
projective linear subspaces, namely hyperplanes. This will be conceptually 
useful when we exploit the projective geometry for the task of object recon- 
struction. 

In Computer Graphics or Computational Geometry, the definition of a poly- 
hedra is based on the underlying data-structure that represents the polyhe- 
dra: a 3D solid especially a polyhedron may be represented by a wire-frame 
model, a surface model or by a volume model, cf. Encarnacao et al. 1997. 

- wire-frame model: A wire-frame model simply consists of a collection of 3D 
edges. There is no connection between the edges defined and no relation- 
ships to existing surfaces. 

- surface model: A surface model is a collection of surfaces, where there is 
no relationship between the individual surfaces. The decision whether a 
point is inside or outside the 3D solid, in our case the polyhedron, cannot 
be made. 

- volume model: A volume model is a complete description of a three- 
dimensional object, the decision whether a point is inside or outside can 
always be made. 

In this work we are mainly concerned with wire-frame models and surface 
models. 
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Fig. 1.1. Wireframe, surface and 
volume models. In this work we 
are mainly concerned with wire- 
frame models and surface mod- 
els. 



1.3.3 Automated Reconstruction 

What is the meaning of the attribute “automatic” when doing automatic 
reconstruction? As there are always human beings who will start the process 
and evaluate the final result, we ask this question with the perspective of 
a potential user of the system: what does a user expect from an automatic 
system? It usually depends on the level of her experience. Somebody with 
no or minimal experience in Computer Vision or Photogrammetry does not 
want to be disturbed by technical details such as thresholds or maximal 
precision but only wants to see the final result. On the other hand input 
parameters may be important for an expert: they can be used to control the 
reconstruction process. An expert might also be interested in intermediate 
results to change the behavior of the system. 

A possible definition for the scale of automation is the amount of knowledge 
that is required to use the software. Opposed to the user-side knowledge is 
the “knowledge” that is encoded in the software, which lowers the amount of 
user interaction. We may distinguish between four degrees of scale: (i) fully 
automatic: no interaction needs to be done; (ii) automatic: simple yes/no 
decisions have to be made or a selection of two or more alternatives; (iii) semi- 
automatic: some quantitative parameters have be set by mouse or keyboard; 
(iv) interactive: the user continuously steers the acquisition, as with classical 
photogrammetric stations. 

The acquisition process itself may be subdivided into three parts, where all 
of these three parts are ideally being automated: (a) initialization: specifying 
the input parameters, (b) process: the actual acquisition and (c) assessment: 
characterization of the results obtained by the system. Table 1.1 summarizes 
the different scales of automation with respect to the acquisition process. 

Of course the ultimate goal is a fully automatic reconstruction started by 
pushing a button and finished by a machine-generate diagnosis of the results. 
This is hard to achieve at the moment and for practical purposes it may 
be advisable to include some minimal interaction to an automated system, 
see Heuel and Nevatia 1995, Fua and Brechbuhler 1996. Vice versa, 
one may use the opposite approach and enhance interactive systems such 
that they reach a higher level of automation. 

The discussion of automation does not only hold for complete systems, but 
also for modules that are used within the system: the modules must be capa- 
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Table 1.1. Different degrees of automation for a reconstruction process 



process step ^ 
1 scale of 
automation 


initialization 


process 


assessment 


fully automatic 


pushing the 
start button 


no questions 


self-diagnosis 


automatic 


selection of 
some qualitative 
initialization 
parameters 


very few 
qualitative 
questions, if at 
all 


confirmation 
resp. rejection 
of results 


semi-automatic 


simple 

qualitative and 

quantitative 

input 

parameters 


some qualitative 
and quantitative 
input and 
adjustment 
steps 


checking and 
possibly editing 
results 


interactive 


qualitative and 
quantitative 
parameters, 
depending on 
the complexity 
of the task 


qualitative and 
quantitative 
input and 
adjustment 
steps, 

depending on 
the complexity 
of the task 


permanent 
steering 
throughout the 
process 



ble to assess their own results and give measures of its accuracy. The input 
parameters must be either simple and/or the module can be steered by an 
earlier module in a consistent manner. 



1.4 Geometric Reasoning 

In the previous section we have seen that the models we use in this work 
all have strong geometric properties: the image capturing process is usually 
modeled by a geometric pin-hole camera, in the images we observe polygons 
and finally we are mainly interested in the geometric parameters of the poly- 
hedral object. Although there are other cues in the images that can be used 
for object reconstruction, such as radiometry and topology, we will largely 
concentrate on the geometric properties. 

In this section we formulate the object reconstruction as a problem of inferring 
the most plausible object from the geometric properties of the image and the 
cameras, hence doing geometric reasoning. 
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1.4.1 Reasoning and Object Reconstruction 

According to Russell and Norvig 1995, p. 163, reasoning or inference are 
terms that “[...] are generally used to cover any process by which conclusions 
are reached”. Usually logical inference or deduction is used as the primary 
method of reasoning. In the late 19th century, the geodesist, mathematician 
and philosopher Charles S. Peirce (1839-1914) distinguished between three 
different methods of reasoning: deduction, induction and abduction, cf. 1.2. 
Deduction is a method of reasoning by which one strictly infers a conclu- 



Table 1.2. Examples for three different methods of reasoning, see Peirce 1931. 



abduction 

(generation of 
hypotheses) 


All beans from , Therefore, these 

, . , these beans are , „ 

this bag are , beans are from 

white ^ ^ ^ this bag. 


induction 

(generalization) 


This bean is 

from this bag, These beans are , ^ , . 

, , . , 1 , n . beans from this 

and this bean, all white. , , . 

, , . bag are white 

and this, ... 


deduction 

(logical 

inference) 


All beans from TPese beans are Therefore, these 

ag are from this bag. beans are white, 

white. 



sion from a set of given (and true) facts by employing the axioms and rules 
of inference for a given logical system. Mathematical theorems are usually 
proven using deductive argumentation, because the conclusion is analytical 
and uniquely follows from the given facts. 

On the other hand, induction is a method of reasoning by which one infers 
a generalization from a series of instances. This inference is only strict if the 
series of instances cover the whole space of possible instances, therefore being 
a complete induction. 

Finally, abduction is a method of reasoning by which one infers an explana- 
tion of observations ( “the beans are from this bag” ) given an hypothesis ( “all 
beans from this bag are white”). Obviously, this method of inference is not 
strictly truth-conserving. Peirce argued that in contrast to deduction, abduc- 
tion reveals new insights into reality. Usually the interpretation of abduction 
is now generalized to give not only one but the possible best explanation of 
observations given an hypothesis. 

The problem of object recognition from images can not be classified in one of 
the strict inference methods deduction or complete induction. It starts with 
the fact that a detection algorithm is only able to state a hypothesis that a 
building has to be reconstructed at a particular location, it can not logically 
deduce that there must be an object without much more constraints such as 
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a close-world assumption. The same argument applies for the reconstruction 
of objects, as there is no stringent sequence of inferences that an object has 
only one fixed set of parameters. 

It can be argued that the task of object reconstruction from a set of images 
is essentially a mixture of weak induction and abduction: at first, we have to 
deal with observations from the real world, thus making a generalization of 
a series of instances, which can be classified as inductive inference. Then in 
a second step we have to build hypotheses and generate the most plausible 
explanation from the observations given the set of hypotheses. 



1.4.2 Reasoning within Geometry 

In traditional Artificial Intelligence (AI), one mostly deals with symbolic rep- 
resentation of knowledge, i.e. some data representation. Geometric reasoning 
deals with specific kind of knowledge, namely geometric information. Kapur 
AND Mundy 1989b, p. 3 define the domain of geometric reasoning as “[...] 
an approach for representing geometric and topological properties and [...] 
geometric procedures that manipulate these representations.”. As mentioned 
at the beginning of this chapter they argue that “[...] a better conceptual 
framework for representing and manipulating geometric knowledge can lead 
to implementation designs that provide more adequate separation between 
generic geometric principles and specific constraints of the application.” . 
Traditionally, geometric reasoning was identified with mechanically proving 
theorems in geometry, see Gelernter 1963 and numerous researchers have 
been working on this field, cf. Richter- Gebert and Wang 2000. On the 
other hand, reasoning methods dealing with geometry have been extensively 
used in applications such as robotics and motion planning, computer vision 
and solid modeling Durrant-Whyte 1989. Considering the above defini- 
tion, geometric reasoning in terms of object reconstruction means to extend 
the given geometric knowledge, i.e. the given images, to a 3D model of the 
unknown object. 

In order to accomplish geometric reasoning for polyhedral object reconstruc- 
tion, three important tasks have to be solved: 

- find a way to represent the geometric knowledge in 2D and 3D in a consis- 
tent manner. Specifically we are interested in the representation of points, 
lines and planes. No special conditions or singularities should arise from 
the representation; for example the horizon in an image is essentially a 
mapping of a 3D line at infinity and should be represented in the same 
way as the mapping of any other 3D line. 

- manipulate geometric entities to obtain new entities. This can be done 
either by unique operations such as the intersection of lines or by finding 
an estimate for an unknown object given a set of observations. 
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- deriving hypotheses of relations between geometric entities that are relevant 
for the application, for example the incidence relation of a line and a plane. 

To restate the relation of reasoning and object extraction in the context 
of geometry, there are two inference methods that can be applied: first use 
induction, e.g. by testing relations between observed instances of points and 
lines and therefore establishing hypotheses of the underlying structure. In a 
second step we may use abduction, for example estimating a line given some 
previously hypothesized configuration of points and lines. The estimation of 
a geometric object can be performed by a least-squares method, which can 
be interpreted as finding the best explanation for the given observations. 

1.4.3 Uncertain Reasoning 

As explicated above, measurements of the real world are inherently uncertain: 
the observed error consists of an abstraction error due to the underlying 
model and an error in precision due to the imperfectness of measuring devices. 
A system that aims to infer new information from given measurements has 
to be able to represent and account for the effects of uncertainty. 

An example might be given as follows: a robot has a mounted CCD camera, 
from which it infers information about its environment. The robot must be 
able to deal with uncertainty since it knows its position and the position of 
other objects only up to some error, so it is not certain about its position. 
Here, dealing with uncertainty means to (i) represent the errors of its position 
in an efficient manner, (ii) propagate these errors when moving the camera 
or changing the position and (iii) taking them into account when making 
decisions, such as ’’Can I go through this door? Will I miss the door when I 
continue moving in the current direction?” . 

Recently, more and more researchers acknowledge that precise logic alone is 
not capable of solving these and many other tasks in a robust way, cf. Mum- 
FORD 2000. In our application, we require that uncertainty should be an 
intrinsic part of geometric descriptions, cf. Durrant-Whyte 1989. 



1.5 Previous Work 

We categorize previous work that relates to this thesis in three categories: (i) 
projective geometry, (ii) geometric reasoning and statistics and (iii) polyhe- 
dral object reconstruction. 

1.5.1 Projective Geometry in Photogrammetry and Computer 
Vision 

Early references in Photogrammetry for the implicit techniques of Projective 
Geometry can be found in Finsterwalder 1899 and Das 1949. In his short 
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paper, Das describes a form of the collinearity equations with the abstract 
parameters of a projective camera matrix and thus being linear in the image 
and object coordinates. He applied the abstract collinearity equations to clas- 
sical problems of Photogrammetry such as absolute and relative orientation 
of images. The approach of linearization was called DLT by Abdel- Aziz and 
Karara 1971. The relative orientation between two images in the context of 
projective geometry was described by Thompson 1968. Another approach 
for relative orientation using algebraic geometry includes VAN den Hout 
AND Stefanovic 1976, for an in-depth review of work on orientation tasks 
including historical references, see Wrobel 2001. 

The application of projective geometry to the problems of reconstruction 
of a real world scene from a set of uncalibrated images was introduced by 
the Computer Vision Community, a first tutorial introduction within this 
community was written by Mundy and Zisserman 1992. One of the key 
advances was the realization, that a scene can be reconstructed from a set of 
uncalibrated images as all possible reconstructions are projectively equiva- 
lent (Faugeras 1992, Hartley et al. 1992). The remaining task to obtain 
a unique Euclidean reconstruction is to find a one-to-one transformation, 
that transforms (“stratifies”) a projective reconstruction to the desired Eu- 
clidean representations. There exist a number of strategies for stratification, 
see e.g. Faugeras and Luong 2001 for a summary. 

The results using projective geometry beginning in the early ’90s have been 
successfully applied to a number of different applications, such as repre- 
sentation of different camera models (Aloimonos 1990, Faugeras 1993), 
computation of relative orientation of two cameras (TORR and Murray 
1997, Hartley 1995b, Zhang 1998 where the basic work dates back to 
Longuet-Higgins 1981), relative orientation of three views (Spetsakis 
AND Aloimonos 1990, Shashua and Werman 1995, Hartley 1997), 
self-calibration (Faugeras et al. 1992, Zisserman et al. 1995) and struc- 
ture computation (Pollefeys et al. 2000). Probably the best references for 
an overview of the use of projective geometry for vision tasks can be found 
in the recent textbooks by Hartley and Zisserman 2000 and Faugeras 
AND Luong 2001, the latter providing a more theoretical insight into the 
field. 

A large subset of the work on projective geometry mostly centers on transfor- 
mations, including camera transformations, relative orientations, especially 
for the uncalibrated case. In this work we mainly concentrate on the repre- 
sentation and manipulation of geometric primitives. 

One particular interesting part of projective geometry when dealing with con- 
structions of geometric primitives is the representation of projective points, 
lines and planes within the Grassmann-Cayley algebra, cf. Carlsson 1994: 
the geometric primitives are represented by projective subspaces and it can be 
shown that the union and the intersection of these subspaces can be treated 
algebraically. The possibility of using algebraic expressions for geometric con- 
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structions is especially useful for formulating geometric problems involving 
multiple images since we can apply the geometric intuition for join and inter- 
section to seemingly complicated algebraic expressions. Some of the applica- 
tions of the Grassmann-Cayley algebra is the formulation of the trifocal tensor 
(Triggs 1995) or the modeling of cameras Fatjgeras and Papadopoulo 
1998. 



1.5.2 Statistical Geometry and Reasoning 

The work in Computer Vision which attacks problems with a reasoning ap- 
proach is usually classified in the domain of knowledge-based image under- 
standing - a comprehensive survey of this topic is written by Crevier and 
Lepage 1997. One particular advantage of reasoning systems is that using 
a set of algorithms with specified in- and output, a system can be tailored 
specific to its task. One problem when designing such a system is the repre- 
sentation of the uncertainty of the given knowledge. Generally, there exists 
quite a few different mathematical theories of evidence that can help to pro- 
vide reasoning and control to an image understanding system, among them 
are fuzzy approaches (Zadeh 1979) and probabilistic approaches. The the- 
ory of fuzzy sets contains general fuzzy measures, from which the concept of 
probability measures is a subclass (Klir and Folger 1988). 

This work follows a probabilistic approach of geometric reasoning. The actual 
reasoning process could be steered using a Bayesian network Pearl 1988, 
which Sarkar and Boyer 1995 used for geometric knowledge base repre- 
sentation and grouping of geometric primitives of different types. In order tp 
apply these kind of networks, consistent representation of the primitives and 
an exact characterization of the performance of the algorithms is needed. 
Since we restrict our application to polyhedral objects, we are interested in 
linear entities, although there exists work on a probabilistic modeling of gen- 
eral curves, see for example Blake et al. 1998. As opposed to representing 
geometric primitives, an example for the uncertain representation of projec- 
tive invariants (Mlfndy 1992) is the work by Maybank 1995. He develops 
a probability density function (pdf) for cross-ratios in images and defines a 
decision rule based on the pdf. This may be extended to more complicated 
invariants. 

PORRILL 1988 developed a system called GEOMSTAT that uses a for- 
malism for a generic statistical estimation using observed points and line 
segments by using a the most convenient representation for the parameter 
vectors and an unconstrained or minimal representation for uncertainty of 
the parameters. The uncertainty is modeled by a Gaussian distribution. To 
estimate unknown objects, a Gaufi-Markov estimation model is used. The ap- 
plication of the system is the automatic reconstruction of wire-frame models 
from images. 
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Durrant-Whyte 1989 describes the uncertainty of entities by a point in 
parameters space with its pdf. Manipulating an entity means manipulating 
the underlying probability density function. He suggests a Gaussian approxi- 
mation of this pdf. He also requires that the parameter representation of the 
entities should be statistically well-conditioned, meaning that there should be 
a one-to-one relationship between the parameters and the entity and that the 
parameters should change smoothly with smooth motions of the geometric 
entity. 

Collins 1993 investigates the probability of projective objects and proposed 
to use a spherical distribution, the Bingham pdf. It is a second-order approx- 
imation to any antipodally symmetric density functions and thus serves a 
similar role on a sphere than a Gaussian on a plane. Errors of entities can be 
propagated, and it is shown how to perform statistical inference on incidence 
and equality relations with a Bayesian approach. These theoretical results are 
applied to grouping of 3D line- and plane-orientations from observed image 
features. 

Seitz and Anandan 1999 introduce a technique for representing points, 
lines and planes as probability functions. In particular, they use the Gaus- 
sian density function and develop a consistent implicit matrix representation 
containing mean and second moments. Then intersections and transforma- 
tions of the entities can be formulated as unique operations on the implicit 
representation and therefore directly transforms the uncertainty representa- 
tion. This can be applied in a consistent way for all types of entities. However 
there is no apparent way to extend this work for join operations. 

Clarke 1998 describes homogeneous covariance matrices for homogeneous 
vectors, which in turn are representatives of projective entities. He shows that 
it is feasible to apply first-order error propagation using these covariance ma- 
trices to obtain uncertainty information for constructed entities such as a 2D 
point from two 2D lines. This has been shown using Monte Carlo simulations 
to compare statistical and predicted covariance matrices. 

Criminisi 2001 presents an approach of using one or more images as a mea- 
suring device based on projective geometry. From simple situations such as 
measuring planar surfaces from one image, he continues with more compli- 
cated three-dimensional measures and using multiple images instead of only 
one. As a measurement without a statement about its precision is practically 
useless, the uncertainty of the final result was computed from the uncertainty 
of the input data by first-order error-propagation. It was demonstrated that 
the assumption of a Gaussian density function is valid for a sufficiently precise 
prediction of the errors of the result. 

To our knowledge, the work by Kanatani 1996 is probably the most gen- 
eral approach to the statistical estimation and testing of three-dimensional 
structures based on image data and geometric constraints. It includes points, 
lines in 2D and 3D and planes in 3D and demonstrates (i) how to correct 
these primitives to satisfy geometric constraints (including testing these con- 
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straints), (ii) optimal fitting of unknown primitives given some observations 
with applications to vision tasks and (iii) derives a criterion for selection of 
possible statistical models for the geometric data. Our work is similar to the 
first two of the three mentioned topics, but motivated by the Grassmann- 
Cayley algebra: the formulations of the testing and estimation problems be- 
come much simpler. 

Finally it is worthwhile noting that the classical technique of bundle adjust- 
ment, extensively used in Photogrammetry, is acknowledged by the Computer 
Vision community to be the optimal general method for fitting cameras and 
structure, cf. Triggs et al. 2000, though the use of robust estimators is 
preferred in the vision context. 



1.5.3 Polyhedral Object Reconstruction 

One of the first works in polyhedral object reconstruction is the one by 
Roberts 1965. He introduced a data driven approach to identify and re- 
construct the pose of polyhedral objects from digital images. At first, image 
features are detected based on the image intensities and then the best model 
in his database is selected that fits to the image features. Motivated by this 
example, researchers investigated the nature of line drawings of polyhedra 
with the hope to obtain useful results for recognition and reconstruction. 
Most notably is the work by Clowes 1971, Huffman 1971 and later SuGi- 
HARA 1986 who developed methods for consistent labeling of polygonal edges 
that referred to the 3D topology. While this work is useful for line drawings, 
cf. Braun 1994, unfortunately it seems that it is quite hard to apply the 
reasoning to real imagery. This is mainly due to the fact that it is almost im- 
possible to obtain accurate topological information from segmentation results 
of complex images. 

Later on, the approach of fitting known polyhedral models to one or more 
uncalibrated images was refined for example by Lowe 1991. The main prob- 
lems with such an approach are: (i) a database of possible models has to exist 
and (ii) the search for selecting the best model can be quite involving. Thus 
one tries to first find parts or invariants of polyhedra such as in Rothwell 
AND Stern 1996 or Mohr et al. 1992. 

An important application of polyhedral object reconstruction is the recon- 
struction of man-made structures such as buildings from a set of aerial images. 
Here the advantage is that usually one has quite precise camera parameters, 
which can be used in the system. For an overview of recent work in this area, 
see Mayer 1999. As mentioned above, a polyhedron is not a building model 
per se, the semantics of its parts are missing: for example surfaces have to 
be labeled as roof surfaces. Still a lot of researchers use polyhedra as the 
underlying geometric model and build their semantic models upon it. 
Reconstruction of buildings may not only rely on gray scale images, one can 
also use laser-data as in Weidner 1997, Brunn and Weidner 1998 or 
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radar data cf. Stilla et al. 2001. But this kind of range-data was found 
to be especially suited for the detection task of buildings by discriminating 
possible areas with the underlying height information, see Weidner and 
Forstner 1995 or Jaynes et al. 1997. 

An automated building detection and reconstruction system based on rect- 
angular blocks was introduced by Lin et al. 1994, where only one image is 
used and extracted features are grouped to higher aggregates until a build- 
ing can be hypothesized. This work has been extended to multiple views, 
see Noronha and Nevatia 2001, where the matching between the images 
is done on the feature level (line segments, junctions) and aggregate level 
(Parallels, Corners, U’s). 

More general classes of polyhedra than only rectangular shapes are treated 
for example by Bignone et al. 1996, who match 2D line-segments with 
color attributes to compute 3D line segments, that in turn are grouped to 3D 
surfaces, for which boundaries are sought in the images. In a last step, the 
surfaces are aggregated to polyhedral parts. Moons et al. 1998 use a similar 
approach, but build the 3D surfaces using a triangulation of image edges, 
that are grouped and matched to complete surfaces using a geometrically 
and radiometrically steered region growing technique. 

Baillard et al. 1999 first compute matching line segments in multiple im- 
ages based on image correlation and geometric constraints, see also Schmid 
AND ZiSSERMAN 1997. Then they determine planar patches for each line 
by finding a projective transformation of image neighborhoods. Then they 
group 3D lines within these half-planes and close them by plane intersec- 
tions. A different approach for 3D grouping of lines and corners solely based 
on geometrical and topological features is presented in Heuel et al. 2000. 
An important last step of a reconstruction system is the validation, that 
the chosen model actually fits to the image data. Some verification methods 
are using shadows Huertas et al. 1993 or walls Noronha and Nevatia 
2001. The verification of building hypotheses based on a set of models and 
the selection of the best model using a minimum description length (MDL) 
criterion can be found in Kolbe 1999. 

Finally, we want to mention some of the user-assisted or semi-automatic ap- 
proaches for polyhedra reconstruction, which do not aim at a full automation 
level. Within the photogrammetric area, there exists e.g. CyberCity Mod- 
eler (Grun and Wang 1999) and the inJECT system (Gulch et al. 
2000), which will be introduced in chapter 5. A system that was presented 
to the Computer Vision and Graphics community is Facade cf. Debevec 
et al. 1996, which especially aims on photo-realistic reconstruction. It uses 
parametrized models, that are interactively specified by the user with the 
aid of a precomputed image segmentation result. The final model is then 
optimized by a minimization function. There are two commercial systems 
with a similar purpose, ImageModeler by Realviz (Realviz 2002) and 
PhotoModeler (Photomodeler 2002). 



1.6 Overview of Thesis 



17 



1.6 Overview of Thesis 

This work presents a unified approach for representing, eomhining and esti- 
mating points, lines and planes in 2D and 3D taking uncertainty information 
into account. To demonstrate the feasibility of the approach the derived meth- 
ods are applied to the task of reconstructing polyhedra from multiple views. 
We use recent advances in the application of projective geometry to derive a 
simple algebraic framework for the representation, construction and expres- 
sion of relations of simple geometric entities such as points, lines or planes. 
An important aspect is the explicit incorporation of uncertainty information 
to this framework. 

Chapter 2 describes important aspects of projective geometry that are nec- 
essary for our task and introduce a representation for points and (infinite) 
lines in 2D and 3D and planes in 3D. These geometric entities will later 
be used to describe polygons and polyhedra. Furthermore basic geometric 
transformations between the entities are described. 

Chapter 3 describes how to manipulate the introduced entities and how to 
establish relations between them: we start with the two basic operators of the 
Grassmann-Cayley algebra, join and intersection, and give simple bilinear al- 
gebraic expressions for them. We analyze the nature of these expressions and 
from there derive simple formulas for testing on incidence, identity, paral- 
lelity and orthogonality. Finally, an estimation scheme for unknown entities 
is presented based on a set of observations related to the unknown. This 
scheme can also be used for estimation of unknown transformations, yielding 
a general formulation for direct linear transforms, DLT. 

Chapter 4 deals with the incorporation of uncertainty information in the alge- 
braic framework of the last chapter. We propose an approximate uncertainty 
representation using a Gaussian pdf and analyze the error that is caused by 
the Gaussian assumption. It turns out that the approximation is valid un- 
der certain well-defined circumstances and enables us to formulate statistical 
versions of the methods from chapter 3, which are bundled in a system of 
procedures for statistically uncertain geometric reasoning, called SUGR, see 
section 4.7 on page 147. 

Chapter 5 validates the usefulness of the SUGR-system by applying it to the 
task of polyhedral object reconstruction from multiple images. We demon- 
strate how to include the techniques both for user-assisted reconstruction 
systems or automated reconstruction systems and show some results. In par- 
ticular it is demonstrated how SUGR can bridge the gap between interactive 
and automated approaches. 

Finally we conclude with a short summary and an outlook for future work. 



2 Representation of Geometric Entities and 
Transformations 



A necessary step of developing a system for geometric reasoning is the choice 
of the representation of the geometric objects that are going to be used. In 
section 1.3 we restricted the possible shapes of objects to polytopes. Poly- 
hedra and polygons may be described by linear subspaces namely points, 
(infinite) lines and planes. In this chapter we want to explore a mathematical 
description of points, lines and planes in 2D and 3D, which from now on are 
referenced as basic geometric entities or just geometric entities. 

We use results from projective geometry, which have been successfully applied 
to Computer Vision in the last decades. This chapter summarizes some of 
the aspects of projective geometry that are important for our task. Note 
however that the notation is simplified compared to a rigorous treatment as 
in Faugeras and Luong 2001, that is too verbose for our purposes. 
Geometric entities will be represented as projective subspaces, which in turn 
can be parametrized by homogeneous vectors. An important observation in 
this chapter is that all homogeneous vectors can be consistently subdivided 
into a homogeneous and a Euclidean part, as proposed in Brand 1966 who 
described the distinction without the explicit context of projective geometry. 
After discussing the homogeneous representation of geometric entities, some 
task-relevant transformations are reviewed, such as homographies and pinhole- 
camera transformations. Finally the duality-principle of projective geometry, 
that will be used in the subsequent chapters, is introduced. 



2.1 Projective Geometry 

When dealing with images of the real world, we are naturally exposed to 
elements at infinity: for example the horizon is an image of infinitely distant 
space points^, which can be easily seen by images taken in flat environments: 
parallel roadsides meet at infinity. Unfortunately Euclidean geometry is not 
sufficient to represent infinite elements and therefore it is hard to deal with 

^ To be precise, this would be true if the world was planar. But the real, almost 
spherical world can be locally approximated (or modeled) by a plane if we mea- 
sure buildings and garages within a small region. 

S. Heuel: Uncertain Projective Geometry, LNCS 3008, pp. 19-45, 2004. 

© Springer- Verlag Berlin Heidelberg 2004 
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them within the Euclidean framework. Therefore we need a more general 
framework which enables us to deal with infinite points, this mathematical 
framework is known as projective geometry. 

In this and the following chapters, we will see that projective geometry pro- 
vides a simple and consistent way to represent and transform geometric enti- 
ties, that are important for Euclidean polyhedral reconstruction. One draw- 
back though is the fact that properties such as parallelity or orthogonality 
can not be directly used; more precisely, they are not invariant with respect 
to projective transformations. The study of invariance of properties with re- 
spect to certain groups of transformation may be regarded as a definition of 
geometry, as Klein 1939 mentioned in his Erlanger Programm. We will come 
back to the issue of invariance in section 3.3 on page 70, where we investigate 
possible relations between geometric entities and their invariance properties 
with respect to Euclidean, similar, affine and projective geometry. We will 
see that for our application it is possible to infer parallelity and orthogo- 
nality of lines and planes from homogeneous vectors by using the Euclidean 
interpretation thereof. 

For a detailed and more precise introduction to projective geometry in the 
context of Computer Vision, see Faugeras and Luong 2001 or chapter 23 
in Mundy 1992. 

2.1.1 Projective Space and Homogeneous Coordinates 

Generally, a projective space P” is the set of one-dimensional vector sub- 
spaces of a given vector space Each one-dimensional vector subspace 

represents a projective point x with the projective dimension 0. 

Assuming n = 1, the projective space is the projective line P^ and is visualized 
in figure 2.1(a) on the facing page: the projective points xi /2 G P^ can be 
identified with lines going through the origin of the vector space K^. Thus 
these lines are one-dimensional subspaces of Note that no coordinate 
system but only the origin is defined here. 

A definition of a projective point x € P" may be given as follows: 

Definition 2 (Projective point) A projective point x G P” o/ dimension 
n is an element of the quotient \ 0 and the equivalence relation =: 

x = y 3Ayf0,x=Ay (2.1) 

In other words: all vectors Ax'* = (Aa:i, . . . , Aa:„, A)^ € A yf 0 are 

equivalent with respect to = and define a projective point x. In other words, 
a projective point x G P" is defined by representing it with a n+ 1 vector x'* G 
the so-called homogeneous vector representation. This representation 
is overdetermined: the equivalence relation states, that the lengths of the 
vectors x'*,y'* G do not matter, only their directions. 
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(c) (d) 

Fig. 2.1. Different interpretations of the projective line P^: in (a), projective points 
xi,X 2 are interpreted as lines going through the origin without defining any co- 
ordinate system. In (b), the projective points are homogeneous vectors, that are 
normalized to the sphere S^. In (c) the homogeneous vectors intersect the Eu- 
clidean axis and define the points xi and X 2 on the Euclidean axis. The role of a 
hyperplane 1 can be visualized by considering the projective plane P^ in (d): two 
projective points xi and X 2 define two points xi,X 2 on the Euclidean plane R^. 
The first point xi is incident to a line I on the Euclidean plane. The line I in turn 
is defined by the homogeneous vector 1 G R® ~ P^, which is the normal of the plane 
that contains the origin of R® and the line on the Euclidean plane R^ . As the first 
Euclidean point xi is incident to the Euclidean line, the homogeneous vector 1 is 
orthogonal to the homogeneous vector xi, or l^xi = 0. 



To simplify the notation, we identify projective points x G P” with their (non- 
unique) coordinate vector representation based on x^ G i.e. x = x^. 

Therefore we are not strict with respect to mathematical notation and do 
not distinguish between projective points and its coordinate vectors as in 
Faugeras and Luong 2001, p. 78. Hartley and Zisserman 2000 take 
a similar approach, see p. 3f. In the following, we assume that the correct 
interpretation is clear from the context. 

One can normalize a homogeneous vector x^ := to avoid ambiguities, 
see fig. 2. 1(b). The index s in the homogeneous vector x^ indicates that the 
vector is normalized and thus lies on the unit sphere. We could use the 
normalization of a homogeneous vector with its length for an alternative 
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definition of the equivalence class = for two homogeneous vectors x, y G 

\ 0 : 



x = y 



V|x| |y| “ |x| 




— = ±^ 
|x| |y| 



( 2 . 2 ) 



We sometimes denote the spherical normalization by Ns(x) = |^. 

The definition of the equivalence class = is ambiguous as the orientation of the 
normalized vectors can be positive or negative. In other words, a projective 
point from P" has two representations on a sphere 5”"+^. To overcome this 
ambiguity one may use oriented projective geometry, as proposed by Stolfi 
1991, where every element has an intrinsic orientation. This is analytically 
expressed by the specialized definition, that homogeneous vectors are identi- 
cal if they differ by a positive scale factor. But the ambiguity is not critical 
for our application, so we will keep the definition in equation (2.2). 

The link between an original Euclidean space K" and the projective space 
P" is now quite easy: a simple mapping K" — s- P" of the Euclidean space to 
the projective space can be written as follows: 

X = (Xi, . . .,XnV e K” ^ Xe = (xi, . . . ,X„, 1)''' G P" (2.3) 



We will denote vectors from R" with italic bold letters such as x, y, whereas 
homogeneous vectors from P" R("+^^are denoted with upright bold letters, 
x,y. 

The index e in the homogeneous vector Xe denotes that we can directly 
extract the Euclidean information from the homogeneous vector. It is a unique 
member of the equivalence class defined by x = Xg. In figure 2.1(c) one may 
interpret the fixing of the last coordinate of the homogeneous vector to 1 as 
follows: take the line parallel to the first axis, intersecting the second axis in 
1. This line represents the real axis on which the Euclidean numbers Xi /2 
are fixed. Connecting the origin with Xi /2 on the constructed line yield the 
projective points. 

For the inverse direction of (2.3), any projective point x G P" (except of ideal 
points) can be mapped to an euclidized version, see also figure 2.2: 

X = (xi, . . . ,x„,x„+i)'^ G P” ^ Xe = l/x„+i (xi, . . . ,x„, 1)"^ G P” (2.4) 



For lower dimensions, we will replace the coordinates xi,X 2 ,xa with the let- 
ters u,v,w to avoid overloaded use of indices. 



Linear Transformations of Projective Points A linear transformation H from 
pra pm represented as a (m -|- 1) x (n -I- 1), such that for x G P" and 

x' G P™: 

x' = Hx (2-5) 

If n = m and the matrix H is regular, we call H a homography. 

Properties that remain invariant under a transformation H are called projec- 
tively invariant. Examples for such properties are collinearity and incidence. 
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transfer 



euclidean 




projective 




homogeneous 


entity 


► 


entity 




coordinates 


a; G M” 




x G P” 




x G 



normalization 



Fig. 2.2. Transfer between Euclidean and Homogeneous Coordinates: giving a 
Euclidean point x € R" we can derive a projective point x G P" which we represent 
as a homogeneous vector x G R"^^, see also equation (2.3). Given a homogeneous 
vector X G R""*"^, we may obtain a Euclidean representation by normalization, 
yielding x G R"*, equation (2.4). In general, the normalization is done by dividing 
the homogeneous vector by its homogeneous part as described in section 2.2. Note 
that we use the symbol x both for a projective point and for homogeneous vectors 
representing the projective points. 



2.1.2 Hyperplanes 



A point in projective space can be regarded as the smallest linear subspace 
that is non-zero. Opposed to a point, a hyperplane is the largest linear sub- 
space that is not equal to the complete space. Examples for hyperplanes are 
lines in 2D and planes in 3D. 

In general, a hyperplane h in P" is defined by n linearly independent points 
yi, • • • ,y„: for example in P^, two points yi,y2 define a line and in P^ three 
points Yi, Y2, Y3 define a plane. Any additional point x is incident to the a 
hyperplane h(yi , • • • , y„) if the determinant of all concatenated homogeneous 
vectors is zero 

lyiyz • • - yn x| = 0 (2.6) 

This is a generalization of the well-known case when checking whether three 
2D points are collinear Bronstein et al. 1996: 



Vi, 1/2 and X are collinear 



Vl V2 ® 
1 1 1 



= 0 



(2.7) 



which can be generalized to the to the 3D case when checking whether four 
points are coplanar: 



1^3 

1 




( 2 . 8 ) 



Yi, Y2, Y3 and X are coplanar <G> 



Yi 

1 



Y2 

1 



(2.9) 
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An alternative definition of a hyperplane can be obtained by using the Lapla- 
cian expansion on the determinant in equation (2.6): if we expand by the last 
column, we obtain n — 1 minors that depend on the vectors yi, • • • y™. 

With hi = (— equation (2.6) is equivalent to 



Both (2.6) and its equivalent equation (2.10) is a necessary and sufficient 
condition to check whether a point x is incident to a hyperplane, x € h. 
Since the minors hi with f = 1, • • • ,n uniquely define a hyperplane, we can 
identify them with the coordinates of a hyperplane, defining the vector h = 
(/ii,--- ,hnY ■ The coordinates hi are called Pliicker coordinates, see also 
section 2.2.4. The vector h is a homogeneous vector as (2.10) is a homogeneous 
equation. As for projective points, it is assumed that it is clear from the 
context whether a homogeneous vector is interpreted as a projective point or 
a hyperplane: in the following lines in 2D are usually denoted by 1, m and 
planes in 3D are denoted by A,B opposed to points denoted by x,y resp. 



It is important to note that h is actually a linear form of since each 

coordinate is a function of the form — > M. Thus the vector h represents 

an element of the dual space (R"+^)* or equivalently (P")*. 

Given a linear transformation x' = Hx for projective points as in (2.5), 
one may be interested in the corresponding transformation for hyperplanes. 
H is a linear mapping from P" to P™, its transpose is defined as the 
mapping from (P™)* to (P")*. With h' G (P™)* being the imaged hyperplane 
of h G (P”)* with respect to H, one obtains 



If H is a homography, thus n = m and H being regular, one can prove (2.11) as 
follows: given x' = Hx, we are looking for the corresponding transformation 
h' = Jh, such that for all y G h and y' G h': y' = Hy. Thus the following 
equation must hold 



This can be true only if J = H 

2.2 Representation of Geometric Entities 

This section introduces the basic geometric entities, that will be used through- 
out the rest of this text. As we want to reconstruct polyhedral objects, it is 
sufficient to deal with geometric entities that can be modeled as linear sub- 
spaces, cf. section 1.3.2 on page 6. Therefore only points, lines and planes 




( 2 . 10 ) 



X,Y 



H^h' = h 



( 2 . 11 ) 



y'"^h' = x'"^Jh = = 0 
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in 2D and 3D are covered. They are represented using homogeneous vectors, 
but we will give a Euclidean interpretation for the vectors of every entity. 
At the end of this section, we will see that the chosen representations are 
special cases of the Pliicker coordinates, a representation of subspaces within 
a projective space. 



2.2.1 Points and Lines in 2D 

Given a 2D point x = (x, y)^ G on the Euclidean plane, one may obtain 
a homogeneous representation using equation (2.3): 





fx\ 




^ wx '' 




fu\ 


X = 


y 


— 


wy 


— 


V 








\ ^ / 




^w) 



see also figure 2.1(d) on page 21. 

A Euclidean representation of a 2D point is straight forward by specifying two 
coordinates, but the Euclidean representation of a 2D line can be manifold: 



slope-intercept form 
axis-intercept form 
angle-distance form 
homogeneous form 
determinant form 
point-angle form 



y = mx + b 




X cos{tp) + y sin{(fi) = d 
ax + by + c = 0 



Xi X2 X 

yi V 2 y 
111 



= 0 



{x - xo) cos(v?) + {y- Vo) sin{‘p) = 0 



Some of these representations do not cover all possible cases (e.g. vertical 
lines for the slope-intercept form) and some require more parameters than 
necessary: the determinant form requires two points Xi, X 2 hence four pa- 
rameters opposed to the fact that only two parameters are necessary, for 
example angle and distance. 

We will start with the homogeneous form a x+by+c = 0, then the parameters 
can be written in a homogeneous vector 






1= b 



V c/ 



If we multiply the homogeneous form with a scalar w and use the homoge- 
neous vector X = {wx, wy, w)~^ = (u, v, w)~^ for a point, we obtain 

au + bv + cw = l^x = x^l = 0 



which is the 2D case for equation (2.10) for hyperplanes. 



( 2 . 12 ) 
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A line segment s; is a finite segment of an (infinite) line. It can be represented 
by a triple (1/^ , xjg , /) , the infinite line 1/s , a mid-point x /g of the line segment 
and a length 1. This is equivalent compared to defining two enclosing points 
on the line, cf. definition 1 on page 6. 



2. 2. 1.1 Homogeneous and Euclidean Parts From the homogeneous vec- 
tors for points it is easy to determine the Euclidean distance from the point 
to the origin O: for a projective point x = {u,v,w)~^ , the Euclidean distance 
to the origin is given by d(x, O) = \J{u/wY + {v/wY = /\w\. 

The coordinates of the 2D line 1 = (a, b, c)^ may be interpreted by comparing 
equation (2.12) with the angle-distance form: they are essentially the same 
up to a multiplication factor A; = 1 / + b'^: 













^ cos{p) ^ 


1 = 


b 


= \ j \J a? Eb"^ 


b 


= 


sin{ip) 




U/ 




U/ 




\ -d ) 



As the scalar d refers to the distance of the line to the origin, we obtain 
d(l, O) = |c|/v(?+6^. Note that the angle (p is the angle of the normal of 
the line 1 . 

Motivated by the above observation of computing the distances to the origin 
and as suggested by Brand 1966 p. 51ff., we will divide the homogeneous 
vectors for 2D points and lines into two parts: 



- the Euelidean part depends upon the position of the Euclidean origin O. 
This part of the homogeneous vector will be indexed with O and is the 
enumerator of the fractions d(x, O) and d(l, O) as above. 

- the homogeneous part is independent of the position of the origin O. It 
refers to the denominator of the fractions d(x, O) and d(l, O). 



For a 2D point x and for a 2D line 1 the partitioning yields 





/ \ 




V 


- ; 1 - 


b 




\Xh) 




W/ 


\c/ 



2. 2. 1.2 Normalization to Euclidean Coordinates Dividing the homo- 
geneous vectors by the norm of the homogeneous parts, x/|a:/i| and l/|^/t|, we 
obtain the Euclidean representation of the 2D entities: the Euclidean coor- 
dinates of a 2D point and the angle-distance form of a 2D line. We call this 
division Euclidean normalization: for any homogeneous vector x representing 
a geometric entity, the Euclidean normalization No(-) is defined as 

No(x) := x/|a;,i| (2.14) 

where Xh is the homogeneous part of the entity x. 
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Note that the proposed normalization No(-) is ambiguous with respect to its 
sign: a Euclidean point x = (2, 3)^ may be represented by xi = (4, 6, 2)^ and 
X 2 = (—4, —6,— 2)^. This leads to No(xi) = — No(x 2 ). The same applies to 
a 2D line with ip = 90° and d = 3, where two representations mi = (2, 0, 6)^ 
and m 2 = (—2,0,— 6)^ yield No(mi) = — No(mi), see figure 2.3. 




(a) (b) 

Fig. 2.3. Ambiguous Euclidean normalization of a 2D point x and a 2D line 
m, which can be represented by xi,X 2 resp. mi, m 2 : the normalizations No(xi) 
and No(x 2 ) resp. No(mi) and No(m 2 ) yield different results. We may choose one 
representation where the homogeneous part is positive oriented, see text for details. 



To reach a unique representation, it is possible to demand that the homoge- 
neous part {xg)h of any normalized homogeneous vector Xe = No(x) should 
be positive oriented: for points, this means that the homogeneous scale factor 
Xh is positive, thus for a 2D point x, we skip the norm by No(x) = 'x.jxh- 
For lines, the direction 1^ should be on the upper half of the circle 5'^, thus 
((0, 1)^ , Ifi) > 0, see figure 2.3(b) As for the spherical normalization, the am- 
biguity may be resolved using oriented projective geometry, Stolfi 1991, 
but the ambiguity only affects the Euclidean output of the reasoning process: 
as long as only homogeneous coordinates are used, the sign of the vectors 
is not important. We will not use Euclidean coordinates directly during the 
complete reasoning, see chapter 3, thus the ambiguity can be disregarded in 
our case. 

To summarize, we are now able to transfer back and forth between Euclidean 
and homogeneous representation as in figure 2.2 on page 23. Furthermore, 
we can easily obtain the distances to the origin: 
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d(x,0) = ^ ; d(l,0) = l^ (2.15) 

\Xh\ \lh\ 

In the next sections, we will define Euclidean and homogeneous parts for 3D 
entities, too, and we will see that there both parts of the vector play the same 
role as in the 2D case. 

2. 2. 1.3 Infinite Points and Line As stated earlier, projective geometry 
allows us to represent entities at infinity. This means that the distance from 
the origin is not defined in the Euclidean interpretation, thus the formulas in 
equation (2.15) are not valid anymore. This can only happen when the norm 
of the enumerator, i.e. the norm of the homogeneous part is zero. 

Property 1 If a homogeneous vector is non-zero, then it represents an entity 
at infinity if and only if the norm of the homogeneous part is zero. 

Infinite points are of the form Xoo = (m,u, 0)^. Since Xqo is a homogeneous 
vector, this is a one-parameter family of vectors; the parameter is the direc- 
tion of the infinite point: taking the angle-distance form from equation (2.13), 
an infinite point Xqo lies on a 2D line 1 = \{cos{ip), sin{(f), —d) if and only if 
the direction angle (p is perpendicular to the directional vector (m,u)^. 

The (unique) 2D line at infinity is of the form loo = (0, 0, c)^, and it is obvious 
that all 2D points Xoo at infinity are on the line at infinity. 



Table 2.1. Interpretation of the canonical basis e; as 2D points and 2D lines. 



homogeneous vector 


2D point interpretation 


2D line interpretation 


ei = (1,0,0)^ 


infinite point in ^-direction 


y-axis 


62 = (0.1,0)^ 


infinite point in y-direction 


a:- axis 


63 = (0,0,1)^ 


origin 


line at infinity 



Canonical Examples As examples for points and lines, we look at the canon- 
ical basis vectors = (0, . . . , 1, . . . , 0) and interpret them as 2D points or 

i 

lines, see table 2.1. This will be useful when analyzing homogeneous transfor- 
mations, section 2.3 on page 34 and construction of new entities, section 3.1.1 
on page 49. 



2.2.2 Points and Planes in 3D 

Points in 3D are represented analogously to points in 2D: given a 3D point 
X = {X, Y, Zy G K^, the homogeneous representation is according to equa- 
tion (2.3): 
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A plane in 3D can be represented in several ways, similar to 2D line rep- 
resentations, see table 2.2.1 on page 25. Since 3D planes are hyperplanes 
for as 2D lines are hyperplanes for P^, a plane is represented as a ho- 
mogeneous vector A = (A, B, C, D)~^ , resembling the homogeneous form 
AX + BY + CZ + D = 0 with a Euclidean point X = {X,Y,Z). Again 
the homogeneous form can be rewritten with a homogeneous point vector X: 

AU + BV + CW + D = A^X = X^A 



Since we deal with polyhedral objects, a plane segment or a planar patch can 
be defined by its polygonal outline: a planar patch P = (Ap, Ipi, . . . , Ipn) is 
a planar region enclosed by n 2D lines \pi,i = 0 . . . n which lie on the plane 
Ap, cf. definition 1 on page 6. 



2. 2. 2.1 Homogeneous and Euclidean Parts We can transfer the defini- 
tion from section 2.2.1 and define the homogeneous and Euclidean parts of a 
point X and a plane A 



X = 



U ' 

V 


fXo] 


A 


A ’ 
B 


w 


UJ 


5 ^ 


C 




The Euclidean normalization of X is then No(X) = X/Xh and No(A) = 
A/\Ah\. Similar to the 2D case, the ambiguity in the sign of Ag = No(A) is 
resolved by testing whether the normalized direction {Ae)h is on the upper 
half of the sphere S'^, thus ((0,0,1)^, {Ae)h) > 0. The Euclidean distance 
d{-,0) to the origin is 



d(X,0) = ^ = |No(X)| ; d(A,0) = ^ = |No(A)| (2.16) 



2 . 2 . 2. 2 Ideal Poiuts aud Plaue aud Cauouical Examples With prop- 
erty 1 on page 28 it is clear that the infinite point Xoo = {U, V, W, 0)^ lies 
on the infinite plane Aqo = (0, 0, 0, 1). The direction of an infinite point Xoo 
is determined by Euclidean part (Xoo)o- 

The interpretation for the canonical vectors Ei,i = 1,...,4 are listed in 
table 2.2, it is a direct generalization of table 2.1 on the preceding page. 
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Table 2.2. Interpretation of the canonical basis Ei as 3D points and 3D planes. 



homogeneous vector 


3D point interpretation 


3D line interpretation 


El = (1,0, 0,0)"^ 


infinite point in ^-direction 


j/ 2 -plane 


E2 = (0,1, 0,0)"^ 


infinite point in y-direction 


® 2 -plane 


Eg = (0,0, 1,0)'^ 


infinite point in z-direction 


a;j/-plane 


E4 = (0,0,0,!)'^ 


origin 


plane at infinity 



2.2.3 Lines in 3D 



A 3D line is different from the previous entities since it is neither a point 
nor a hyperplane. It is rather a one-dimensional subspace in 3D which can 
be represented by two (Euclidean) points Xq and Yq, cf. figure 2.4(a). One 
needs four parameters to represent a line in 3D, as shown in figure 2.4(b): 
assume two planes have been chosen in advance such as the coordinate planes 
on the XY respectively XZ axes. Then the two intersection points of the 3D 
line and the planes can be described by two parameters each. 

There exists a number of other representations for 3D lines, but in the follow- 
ing we will derive a homogeneous vector representation based on two given 
points, see figure 2.4(a): a 3D line refers to a homogeneous 6- vector 



L(Xo,Yo) 



which consists of two parts: 



/Yq- Xq\ 
V^o X YoJ 




- the first part Lh contains the direction Lh = Xq — Xq of the line L and 
has 2 degrees of freedom. Note that this part is the homogeneous part of 
the 3D line vector as it is independent from the origin. 

- the two points Xo,Xg and the origin O define a plane, whose normal is 
given by the cross-product Lq = Xg x Yq- With the direction of Lq we 
have fixed the line L(Xo,Xq) in a specific plane. Fixing the plane adds 
only one additional independent parameter to the existing two of the first 
part Lh, since Lq and Lh are constrained to be perpendicular to each 
other, 

LlLh = 0 (2.17) 

This constraint is called the Pliicker constraint of a 3D line, see below for 
more details. 

In order to uniquely fix the 3D line we need the minimal distance d(L, O) 
of the line to the origin as the fourth parameter of the 3D-line. d(L, O) is 
equivalent to the altitude h of the triangle (O, Xg, Y g). Because the norm 
\Lo\ of the cross-product Xg x Xg yields twice the area of the triangle 
and \Lh\ is the length of the base of the triangle, we can write 
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d(L,0) = |^ (2.18) 

Thus the distance as the fourth parameter the 3D line is implicitly given by 
Lo- The 3- vector Lq = Xq x IFq completely depends on the distance of the 
origin and thus Lq is the Euclidean part of the 6-vector L = {Lh, Lq)~^ ■ 
The Euclidean part vanishes, i.e. Lq = 0 , if and only if the line intersects 
the origin. 




(a) (b) (c) 

Fig. 2.4. A 3D line may be determined by two points which intersect with two 
planes of the coordinate system; the two points have 2 degrees of freedom since 
they are fixed to planes (left). Two Euclidean points Xq and l^o define a 3D line 
(middle). In this work we choose two 3- vectors for the representation of L: the 
direction of the line Lh = Xo — and the 3-vector Lo ~ Xo x Yo- We assume 
Lo to have the length equal to the distance of the 3D line L to the origin (right). 



To summarize the above analysis, a 3D line is represented by a homogeneous 
6-vector L = (Li, L2, T3, L4, L5, Lg)^, where the homogeneous part Lh = 
{Li,L 2 ,L'iY is constrained to be orthogonal to the Euclidean part Lq = 
, i.e. L^Lo = 0. Thus the 6-vector L has 4 degrees of freedom, 
considering both the orthogonality and homogeneous constraint. As in 2D, 
a line segment Sl is a finite fragment of L and is represented by a triple 
(L,X/s,T), where is the mid-point of the line segment and L is the 
length of the segment. 

The Euclidean normalization of a 3D line Nq(L) = L/|T;,| uses the for- 
mula already known from points and hyperplanes. The ambiguity in the 
sign of the homogeneous part (Te)/t of Lg = No(L) is resolved by decid- 
ing whether the normalized direction (Lp)h is on the upper half of the sphere 

«0,0,iy,(L.),0>0. 
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Some Examples Again, we look at some canonical examples. A line L = 
{Lh,Lo)~^ lies at infinity if the homogeneous part vanishes, Lh = 0. Thus 
a line at infinity is defined by the normal direction Lq and has 2 degrees 
of freedom. Note that a line at infinity can only be computed using equa- 
tion(2.19). 

Taking the canonical 6- vectors E^, we obtain the coordinate axis and three 
distinct lines at infinity, namely the lines lying on the canonical planes of the 
coordinate system, cf. table 2.3. 



Table 2.3. Interpretation of the canonical basis Ei as 3D lines. 



homogeneous 6-vector 


3D line interpretation 


El = (1,0,0,0,0,0)'^ 


a:- axis 


Ea = (0,1,0,0,0,0)'^ 


y-axis 


Es = (0,0, 1,0, 0,0)^ 


2 -axis 


E4 = (0,0,0,1,0,0)'^ 


line at infinity on yz-plane 


Eg = (0,0, 0,0, 0,1)^ 


line at infinity on xy-plane 


Es = (0,0, 0,0, 1,0)^ 


line at infinity on x^-plane 



We summarize the representations of points, lines and planes in 2D and 3D 
in table 2.4 



Table 2.4. Homogeneous representation of points, lines and planes in 2D and 3D. A 
line in 3D is represented with Pliicker coordinates, for which the Pliicker constraint 
L^Lo = 0 holds. 





2D 


3D 


point 


= {u,v\w) = (xo,Xh) 


^iU,V,W-,T) = {Xl,XH) 


line 


= ia,b-c) = ill,lo) 


= {Li,L2,Ls-L4,L5,Le) = {LI,LI) 


plane 


- 


= [A, B,C-,D) = {Al,Ao) 



2.2.4 Pliicker Coordinates 

The previous representation of the 3D line is compatible with the repre- 
sentation using the so-called Pliicker coordinates. The geometrician Julius 
Pliicker^ introduced a representation of fc-dimensional subspaces within an 

^ Julius Pliicker, born 1801 in Elberfeld, Germany, died 22 May 1868 in Bonn, 
Germany. Professor of mathematics at Halle in 1834, at Bonn since 1836. 
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n-dimensional projective space based on niinors of a (n + 1) x (fc + 1) 

matrix with given coordinates, cf. Faugeras and Luong 2001. 

An example for Pliicker coordinates was already introduced in section 2.1.2, 
where three points X,Y,Z form a 2-dimensional subspace, a hyperplane. 
From the 4x3 matrix = (XYZ) one obtains 4 minors or 4 Pliicker 
coordinates. 

Assume now a 3D line L which was formed by two points X, Y. It is a one- 
dimensional subspace with and there are = 6 coordinates, which 

are derived from 2x2 minors kj of a 4 x 2 matrix = (X, Y). This matrix 
consists of two homogeneous vectors of 3D points X, Y - we can also use 3D 
planes, as we will see in section 2.5 on page 40. For we obtain 



U = 



/Ai Yi^ 
X2 Y2 


ll2 = 


Xi Yi 
X2 Y2 


I23 — 


X2 Y2 
X3 I3 


I34 = 


X3 Y3 
X4 Y4 


X3 Y3 
\X4 Y4) 


/l 4 = 


Ai Yi 
X4 Y4 


I24 = 


X2 Y2 
X4 Y4 


ll 3 = 


Xi Yi 
X3 Y3 



(2.19) 



If we assume the homogeneous part of X,Y to be fixed as X 4 = Y 4 = 1, 
it is easy to see that the Pliicker coordinates kj directly relate to our 6- 
vector L = '■ Lh = — (^i 4 , ^247 ^ 34 )^ = (^ 417 ^ 427 ^ 43 )^ and Lq = 

(^ 237 —^ 137 ^ 12 )^ = (^ 237 ^ 317 ^ 12 )^- Scaling X,Y with homogeneous factor A 
resp. scales the Pliicker coordinates by A • /r. 

Note that the Pliicker coordinates kj have been expressed according to a par- 
ticular order of the index pair i,j. We are free though to choose other index 
pairs, that means any other combination of 2 x 2 minors. For our choice of 
L = {Li,,Lo) we have (*,j) e {(4, 1), (4, 2), (4, 3), (2, 3), (3, 1), (1, 2)}. This 
has only the effect on the ordering and the signs of the Pliicker coordinates 
within a vector, but makes it possible to distinguish clearly between a ho- 
mogeneous and a Euclidean part. Note that other authors choose different 
Pliicker coordinates for lines, cf. Hartley and Zisserman 2000, Stolfi 
1991. 



Pliicker Constraint In general, the Pliicker coordinates are not independent 
from each other, in fact there is a set of quadratic relations between them. 
For the cases in 2D and 3D space, a quadratic relation of Pliicker coordinates 
only occurs for line in 3D, we obtain only one condition: 



Uiha + U2I31 + ^43^12 — 0 



( 2 . 20 ) 



All 6-vectors L = (/ 41 , ^ 42 , ^ 43 , ^237 ^ 31 , ^ 12 ) satisfying equation (2.20) corre- 
spond to valid 3D lines. As mentioned above the Pliicker constraint (or 
Pliicker condition) is obviously satisfied with the proposed construction as 
L^Lq = 0. Note that the constraint (2.20) defines a manifold in K®, contain- 
ing all valid 3D-lines. If the vector L is known to be spherically normalized, 
the constraint defines a manifold on the sphere S^. 
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Thus the vector notation for lines appears to have a disadvantage compared 
to the vector representation for points and planes: in addition to the homo- 
geneous factor it has another degree of freedom which is constraint by the 
Pliicker condition. This situation can be compared to the fundamental ma- 
trix, see section 2.3.3 on page 36, which is a homogeneous 3x3 matrix with 
7 degrees of freedom. In chapter 3 and 4 we will see that it is easy to cope 
with the over-parameterization in the context of geometric reasoning. 



2.3 Basic Geometric Transformations 

After introducing the basic geometric entities, we now review possible ge- 
ometric transformations on them. One property of these transformations is 
that they at least preserve straight lines and incidence relations. An in-depth 
treatment of such geometric transformations on projective entities can be 
found in Hartley and Zisserman 2000. 

Transformations may be applied to points, lines or planes, though usually 
one refers to point transformations. Reformulating point transformations to 
transformations of 2D lines and 3D planes is mentioned in section 2.5.2 on 
page 44. 

In the following, we concentrate on the following types of transformations: 

- Linear mappings within image space resp. object space, called homogra- 
phies in 2D and 3D with affine transformations and similarity transforma- 
tions as special cases 

- Linear mapping from object space to image space, called projective camera 
transformations with afhne camera transformations as a special case 

- Linear mapping between two different images, called fundamental matrix. 

A more detailed analysis and the application of the transformations are in- 
troduced in section 3.1.2 on page 56. 

Matrix and Vector Representation The transformations will generally be rep- 
resented by a n X m matrix M, such that x' = Mx. If a matrix is a trans- 
formation of homogeneous vectors, then it is a homogeneous transformation 
M = AM as it is defined up to scale, denoted be upright sans-serif letters. 
Alternatively, we will sometimes represent the matrix as a vector m = 
vec{M~^)) where the transposed rows of M are stacked up in a mn-vector, 
see definition B.2 on page 184. 

2.3.1 Homography 

Homographies are projective transformations, that linearly map points within 
a projective space P”. Thus it can be written as a simple matrix- vector mul- 
tiplication 



2.3 Basic Geometric Transformations 



35 



x' = Hx 



with x',x G and the (n + 1) x (n + 1) regular matrix H. 

An affine transformation Ho is a special case of a homography and combines 
translation T, rotation R, the scaling of both axes M and skew W of the 
original space resp. P^. For 2D, we obtain 



H 



a 



WM RT 



1 1 


s/2 


0^ 




^ mi 


0 


0^ 




^ cos(a) 


sin(a) 


0^ 


s/2 


1 


0 




0 


m2 


0 




— sin(a) 


cos(q!) 


0 


V 0 


0 


V 




VO 


0 






1 0 


0 


1/ 


'’hii 


hi2 
















h2i 


h22 


^23 














V 0 


0 


1 

















0 

0 1 -ty 

[o 0 I ) 

( 2 . 21 ) 



As an affine transformation in 2D has six degrees of freedom, all six entries 
in Ho are not constrained. We may constrain the upper left 2x2 matrix 
H'a = to obtain a similarity transformation by H'^ = rri^l. This 

leads to two independent quadratic constraints in 2D. In a similar way, we 
obtain five independent quadratic constraints in 3D. 

In general, the entries of a homography H can be interpreted using the canon- 
ical examples from section 2. 2. 1.3 on page 28: the multiplication He^ with 
the canonical vectors yields the *-th column of the homography H. Thus 
for example the first column {hn, / 121 , is the image of the infinite point 

in the direction of the x-axis. It is obvious that for an affine transformations 
Ho infinite points remain at infinity. 

The first row (/112, /112, ^13)^ of a homography H can be interpreted using 
lines: as H^F = 1, cf. equation (2.11), the first row is the line which is mapped 
to ei. Thus it is easy to see that for affine transformations, the line at infinity 
remains at infinity. 



2.3.2 Projective Camera 

The most common model for cameras is the pinhole camera model: a point 
X in 3D space is projected into an image by computing a viewing ray L'(x') 
from the unique projection center Xq to X and intersecting this viewing 
ray with a unique image plane, see figure 2.5. The image formation process 
described above can also be modeled as a general linear transformation of 
to P^, which again can be written as a matrix- vector multiplication x' = PX, 
where P is a 3 x 4 matrix. 

This time we interpret the rows A^, and of the matrix P = ( | 

\ cj J 

as planes A,B,C and the multiplication x' = PX contains three equations 
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Fig. 2.5. Pinhole camera model and the 
three camera planes A, B and C. 



of the form (2.10) on page 24. If x' = {u,v,w)~^, then u = 0 if and only 
if X G A. Similarly, z; = 0 X G B and u; = 0 X G C, thus a 
point x' lies at infinity if X G C. The three planes intersect in the projection 
center Xq, as the image point is undefined ifX= already observed 

by Das 1949. 

Given the exterior orientation of a camera with rotation R and projection cen- 
ter Xq and the inner orientation with the calibration matrix K, a Euclidean 
interpretation of the projection matrix can be formulated as follows: 



P = K/?[/|-Xo] 



/ 



c 



cs 




with K = 



0 

VO 



c(l -I- to) 

0 



Vh 

1 / 



( 2 . 22 ) 



with skew s, focal length c, principle point {xh, VhY and scale difference to 
between the first and second coordinate axis in the image. 

Finally, as in the last section we can fix the last row of the transformation 
matrix with = (0, 0, 0, 1)^ to obtain a special subclass of projective cam- 
eras: the plane C then lies at infinity and therefore the projection center Xq, 
too. This means that all viewing rays are parallel to each other and we obtain 
the affine camera 

^Pll Pl2 Pl3 Pl4^ 

^ “ P21 P22 P23 P2i 

0 0 0 1 y 

A more detailed description of various camera models can be found in Hart- 
ley AND ZiSSERMAN 2000 Or in Forstner 2001b. 



2.3.3 Fundamental Matrix 

Assume there are two images given, taken with different cameras Pi and P 2 . 
Suppose now a point X in object space is projected in the first and second 
image, x' = PiX and x" = P 2 X, cf. figure 2.6. It is clear that the projection 
center Xqi, Xq 2 and the point X define a plane, the so-called epipolar plane. 
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Now suppose we take two arbitrarily chosen image points x' and x" from the 
first resp. second image. If we want to test whether they are caused by the 
same object point, one has to test whether the two viewing rays L' and L" 
are coplanar. 




Fig. 2.6. Epipolar geometry: two pro- 
jection centers Xoi and Xq 2 are con- 
nected by an epipolar line, which in- 
tersects the image planes at e' and e" . 
The two viewing rays L' and L" point 
to the object point X. 



But one can also apply this test in 2D by projecting the viewing-ray L' into 
the second image, thus obtaining an image line 1", the so-called epipolar line 
of x' which in turn has to be incident to the second image point x". This 
2D relationship between the first point x' and the second point x" can be 
expressed as a coplanarity condition with the 3x3 so-called fundamental 
matrix F: 

x'"^Fx" = 0 (2.23) 

Thus the fundamental matrix relates one image to another image. There are 
many ways to derive an algebraic expression for the fundamental matrix, for 
a review see Hartley and Zisserman 2000 or Faugeras and Luong 
2001^. In section 3.1.3 we will give an explicit formula for computing F given 
two cameras Pi, P 2 - 

The vector defined by Fx" is actually the homogeneous vector for the epipolar 
line \" and by transposing equation 2.23, one also obtains the epipolar line 
F, in summary: 



1' = l'(x") = Fx" 1" = l"(x') = F^x' 

Note that the projection of camera center Xpi into the second image results 
in a point e', which is called epipole; likewise we can obtain the second epipole 
e". For epipoles, there is no epipolar line defined, thus e.g. Fe" = 0 and e" 
being the nullvector of F. 

The fundamental matrix has 7 degrees of freedom: this is due to the fact that 
the 3x3 matrix is homogeneous and singular, i.e. det(F) = 0. Thus there are 
two constraints for the 9 matrix elements. 



^ Note that they use the transpose F^ as a dehnition for the fundamental matrix. 
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If the internal camera parameters Ki and K 2 are known, only 5 parameters 
are unknown and the matrix is called essential matrix, Longuet-Higgins 
1981 and has 5 degrees of freedom. 

2.4 Conditioning of Homogeneous Entities 

Up to now we have not imposed any restrictions on the homogeneous co- 
ordinates of geometric entities nor on the parameters of geometric transfor- 
mations. As the homogeneous entities are created by Euclidean objects, cf. 
figure 2.2, their coordinates are expressed with respect to an existing ref- 
erence coordinate system. In close-range applications, we usually have the 
reference system nearby the entities. But in aerial imagery, the reference sys- 
tem is usually a geodetic system, like the GauB-Kriiger coordinate system 
or its international variant, the UTM system (universal-transversal-mercator 
system), cf. Witte and Schmidt 1995. For example, the tip of the dome of 
Aachen has the coordinates x = 2505940. 53[m], 1 / = 5626590. 37[m]. Embed- 
ding this point in homogeneous coordinates as in equation (2.3) would yield a 
difference of seven orders of magnitudes between the Euclidean part X o and 
the homogeneous part Xh- When considering subsequent computations with 
these coordinates, such a coordinate representation is undesirable in terms of 
numerical stability. 

A solution is to define a local coordinate system that is close to the set 
of given object points, which is advisable to do prior to all methods that 
are discussed in this work. But if the distances between different points are 
large with respect to the units, an additional scaling of the coordinates is 
needed. For homogeneous vectors of geometric entities this means to either 
choose a larger Xh instead of 1 or equivalently to scale down the Euclidean 
coordinates by some factor. Both solutions can be achieved by scaling one part 
appropriately without changing the other part. We call this scaling operation 
eonditioning of a geometric entity. 

One may choose the second option for conditioning, i.e. scaling the Euclidean 
part of a geometric entity without scaling its homogeneous part. This has the 
effect, that the geometric entities of the vectors move closer to the Euclidean 
origin, cf. section 2.2. The conditioning operation can be represented by a 
matrix multiplication x' = W(/)x, where / is the conditioning factor and W 
is the identity matrix except of the diagonal elements, which correspond to 
the Homogeneous part of the vector. The following matrices for 2D and 3D 
entities are obtained: 




(2.24) 
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For scaling down the Euclidean part, one has to Choose / < 1. To go back 
to the original coordinates, one simply applies the inverse of the conditioning 
factor, X = W( j). Given a set of geometric entities, one has to find the entity 




Fig. 2.7. Effect of conditioning of homo- 
geneous vectors: a point x moves closer to 
the M-axis, a line 1 moves closer to the v- 
axis, but the angle between the two new 
vectors remains invariant. 



X that is furthest away from the origin, and choose a conditioning factor 
fmin, such that x' = \N{fmin)^ is Sufficiently close to the origin. Then apply 
the conditioning factor to all other entities. 

For the projective axis, the effect of conditioning to the homogeneous vectors 
is depicted in figure 2.7: for a point x = {u, u)^, the first coordinate is scaled 
by a factor /, the second coordinate remains invariant. The vector is now 
closer to the u-axis, a Euclidean normalization onto would yield a scalar 
closer to zero. For a hyperplane 1 = (a, 6)^, the second coordinate is assumed 
to be the Euclidean part and thus the first coordinate is scaled by /, yielding 
a vector closer to the M-axis. Note that the dot-product between x and 1 
remains invariant when scaling with the same conditioning factor. 

For a transformation, we generally have two different sets of entities x^ and 
x', for which two different conditioning factors / and f' have to be computed. 
A conditioned transformation matrix H° can be obtained by 

H° = W(/')HW^^(/) (2.25) 

The reconditioning of the estimated transformation H° is then given by 

H = W^i(/')H°W(/) (2.26) 

The conditioning operation is not only useful to obtain small coordinates: as 
we are going to see in the next two chapters, conditioning becomes important 
when 

- computing optimal estimations from a set of observations, cf. section 3.4 

- reducing the bias of the statistical representation of uncertain geometric 
entities, cf. section 4.3. 

Algorithm 2.1 shows a method for conditioning of a set of geometric enti- 
ties. The algorithm is used in section 3.4 on page 80 when constructing an 
unknown entity by a set of observations and will turn out especially useful 
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Conditioning a set of geometric entities 

Objective-. Transform a set of lines, points and planes in 3D such 
that they move closer to the Euclidean origin. 

Given are n = nx + n_L + ua entities with nx 3D points Xfc, til 
lines L; and tla planes A„i- 

Additionally, the required conditioning factor fmin < 1 is given, 
i.e. the minimal ratio between the homogeneous part and the Eu- 
clidean part of a geometric entity. 

1. find the maximum ratio maxho of the norm of the Euclidean 
and homogeneous parts for points, lines and planes 

maXho ■= maxij,fe for v £ {Xi,Lj, Aj,} 

2. determine whether it is necessary to scale the Euclidean parts, 
if max/io < /min then 

f . — fmin ' TTLCtX hO 

if / < £M then / := fminimaxo 

with maxo = maxij^k |vo| for v € {Xi,Lj, A*,} 
v° - W(/)v for V £ {Xi, Lj, A4 



Algorithm 2.1: Algorithm for conditioning a set of geometric entities. The con- 
ditioning changes the ratio between the norms of the Euclidean and homogeneous 
part of a vector, which should be smaller than fmin- The machine accuracy cm is 
the smallest number such that 1.0 -I- cm > 1.0. The conditioned entity is denoted 
by v°, for a definition of the matrices W(/), see equation (2.24). 



when discussing uncertainty representations of the homogeneous vectors, see 
section 4.3.3 on page 117. 

Note that because we have subdivided all homogeneous vectors of geometric 
entities into a homogeneous and a Euclidean part, the conditioning of geo- 
metric entities is very simple and works consistently for homogeneous vectors 
for points, lines and planes within a coordinate frame. 



2.5 Duality Principle 

The duality principle is an important feature of projective geometry as all 
propositions occur twice: taking the dual of a true proposition yields a new 
proposition, possibly with a different meaning. As an example for duality, 
consider the property from section 2.1.2: three 2D points xi,X 2 ,X 3 £ are 
collinear iff |xi X2 X3I = 0. It is possible to replace the points with lines, which 
are elements of the dual space (P^)*. Then we obtain the dual property: three 
2D lines Ii,l2,l3 intersect in one point iff |li I2 I3I = 0, see figure 2.8. 

In the following we will discuss dual entities and their transformations. They 
become important for the geometric constructions described in chapter 3, 
where we introduce the duality between the join and intersection of entities. 
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Fig. 2.8. Checking whether three points are collinear (left) is dual to checking 
whether three lines intersect in one point (right), see text for details. The sign 
o— • is used here to express the duality between the two geometric situations. 



2.5.1 Dual Entities 

Dual Points and Hyperplanes As demonstrated by the last example and men- 
tioned in section 2.1.2, points and hyperplanes in 2D and 3D are called dual 
to each other. As they use the same kind of homogeneous vectors, one can in- 
terpret a homogeneous point vector x as a hyperplane: we denote this change 
of interpretation by x and call x the dual of x. The vector x has the same 
coordinates, but now represents a hyperplane instead of a point. 

For example, given a 2D point x = {x,y,iy its dual x is a 2D line. Setting 
1 = X, we obtain the normalized line No(l) = y, 1)^- Thus the 

distance of the line 1 = x is the inverse distance of the point x to the origin. 
Additionally, the point and its dual line lie on opposite sides of the origin, see 
figure 2.9(left). The same reasoning applies to 3D points X, where its dual is 
a plane X, see figure 2.9(middle). 




Fig. 2.9. Visualization of dual entities. Left: the dual x of a 2D point x is a 2D 
line: middle: the dual X of a 3D point X is a plane; right: the dual L of a 3D line 
L is again a 3D line, the distance d of the dual entities is inverse to the distance of 
the original entities. 



Dual Lines One question has not been answered yet: what is the dual of a 
3D line L? To answer this question, we consider again the relationship that 
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was derived between a point and a hyperplane, see equation (2.10). In 3D, a 
point U is incident to a plane A if U^A = 0. If the plane A was derived by 
three points X, Y, Z, we obtain 

U'^A = 0 A'^U = 0 44> |X,Y,Z;U| = 0 (2.27) 

see equation (2.6). The dot-product U^A is actually an instance of the so- 
called cap-product, denoted by < U,A >: in general the cap-product is 
defined as a product of elements of two dual projective subspaces onto K. 
More precisely, the cap-product is defined for the two corresponding vector 
spaces where the sum of the dimensions of the underlying vector subspaces 
is equal to the dimension of the vector space. For example, a 3D point with 
projective dimension 0 has a vector dimension of 1 as the projective point in 
is represented by a line in cf. figure 2.1(d) for the 2D case. The dual of 
a 3D point is a 3D plane with projective dimension 2 resp. vector dimension 
3. Summing up the vector dimensions yields 14-3 = 4 the dimension of M'^. 
To derive a cap-product involving 3D lines, we can re-interpret the determi- 
nant |X, Y,Z ; U| by defining two lines L(X, Y) and M(Z,U): 



|^^;^| = 0 (2.28) 
L M 

This equation holds if the four points are coplanar or equivalently, if the lines 
L and M intersect in one point. We can expand the determinant in (2.28) to 
hold the Pliicker coordinates of the lines L, M: 



Xi Yi Ui 
X 2 Y 2 Z 2 U 2 
X 3 Y 3 ^3 C /3 
Xi Y4 ^4 C/4 



Ai Yi 
Xi Y 4 



^2 C /2 
^3 C/3 



X 2 Y 2 
X 4 Y 4 



^1 C/i 
^3 C/3 



X 3 Y 3 
Xi Y4 



c/i 

Z2 U2 



X2 


Y2 


^1 


C/l 




Xi 


Yi 


Z2 


U2 


+ 


Xi 


Yi 


Z3 


c/3 


X3 


^3 


Z4 


Ui 




X3 


Y3 


Zi 


Ui 




X2 


Y2 


Zi 


Ui 



(2.29) 



= 4-/1477123 — /2477713 4- /3477712 4- /23''77l4 ~ /l3''7l24 4- /l277l34 

= — /4 1 777 2 3 — /42 77731 — 1^^112 ~ /23''Tl41 ~ hl'<Tli2 ~ /l277l43 



The 2x2 minors are the Pliicker coordinates for the lines L and M, see also 
section 2.2.4 on page 32. One can check that equation (2.28) is equivalent to 

-LlMo - LlMh = 0 (2.30) 

Thus we have derived a condition for the incidence of two lines L and M 
from the incidence of a plane A and a point U. 
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Equation (2.30) defines the cap-product < L, M > of two lines in a similar 
way as < A,U > with equation (2.10); the dual of a 3D line is again a 3D 
line. Interpreting the cap-product by a dot-product of dual entities, we obtain 

< L, M >:= -LIMo - L^Mh = -L'^M with M := ( 

V Mh 

where M is the dual of the line M and is obtained by switching the homo- 
geneous and Euclidean part of the vector M. The visualization of a line and 
its dual is shown in figure 2.9(left). Note that again the point and its dual 
line are on opposite sides of the origin and the dual has an inverse distance 
to the origin with respect to the original entity. 

To summarize, the possible cap-products in 2D and 3D are: 



entity 1 


dim. of 
vector 
subspace 


dual 
entity 2 


dim. of 
vector 
subspace 


dim. of 
vector 
space 


cap-product 


point X 


1 


line 1 


2 


l-k2=3 


<x, 1 > := x"''! 


point X 


1 


plane A 


3 


l-k3=4 


<X,A> :=XTA 


line L 


2 


line M 


2 


2+2=4 


<L,M> :=LTM 



Usually the cap-product is defined in the context of the so-called double or 
Grassmann-Cayley algebra, where operations such as join and intersection 
form an algebra on projective subspaces. In the next chapter we introduce 
these operations on the geometric entities defined. Again, for a rigorous treat- 
ment of this subject, see Faugeras and Luong 2001. 

An alternative proof that two lines are incident if equation (2.30) does not 
explicitly use determinants but only analytical geometry and follows Brand 
1966, p. 54: assume a point (A^, 1) G TJ,) and (A^, 1) G = 

(M~[, MJ)). The lines are coplanar if and only if the vectors (A — y), Lh and 
Mh are coplanar, or algebraically (A — YY {Lh x Mh) = 0. The assertion 
follows now from the fact that Lq = A x Lh and Mq = Y x Mh- 

Algebraic Expression for Dual Entities As we want to express all operations 
on entities within linear algebra, we define the dual operator as a linear 
transformation from one projective subspace to another. For points and lines 
in 2D and for points and planes in 3D, the dual operator is the identity matrix 
k resp. k: 

x:=/gx ,I:=/3l ; X:=/4X ,A:=kA 

For lines in 3D, we define the dual operator as follows, considering our choice 
for the Pliicker-coordinates: 
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L := CL 



with C := 




0 k\ 
k 0 ) 



(2.31) 



Note that applying the dual operators I 3 , U and C changes the interpretation 
of a homogeneous vector from a projective entity to its dual. 



2.5.2 Dual of Point Transformations 

The existence of dual entities also has an effect on the transformations men- 
tioned in section 2.3 on page 34, namely homographies, projective trans- 
formations and fundamental matrices. There exists a theorem to derive a 
duality property for linear mappings of all geometric entities. In section 2.1.2 
it was already mentioned that given any linear map H from P" to P™, the 
dual mapping is the transpose that maps the dual spaces (P™)* to (P”)*. 
This behavior can be transferred to any of the proposed geometric entities, 
see Faugeras and Luong 2001, p. 154. 

For example assume a projective transformation P of a 3D-point X to 2D 
point x', the duality relation for x' = PX can be written as 

x' = PX 0 -. P'^1' = A (2.32) 

where again the simplified notation x' = F and X = A holds. The sign o-» 
is used here to express the duality between the two expressions. 

Since we know how to interpret the elements of the dual spaces (P™)* and 
(P")* we can directly apply this to 2D and 3D. In equation (2.32), a 2D 
image line may be back-projected in 3D using a projective camera matrix P 
and results in a projective plane A, which intersects the focal point and the 
2D line on the image plane. 

Similarly, the rules of duality for homographies H, projective transformations 



P and fundamental matrices F are 






x' = Hx 


0 — • 


H^F = 1 


X' = HX 


0 — • 


II 

1 - 

X 


X 

II 


0 — • 


HiL' = L 


x' = PX 


0 — • 


P^^F = A 


F = Fx" 


0 — • 


II 

\— 

LL 


II 

0 


0 — • 


II 

I— 

a 



The last duality relation involves the projective camera transformation Q for 
2D lines, which will be covered in section 3. 1.2.2 on page 57. 

Right now, the representation of the line transformation L' = H^L is not 
obvious as not all homographies V\l : P® ^ P^ within the five-dimensional 
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projective space correspond to valid homographies in P^. We will deal with 
line transformations in section 3.1.2 on page 56: as soon as we have 
introduced algebraic expressions for constructions join and intersection, we 
are able to derive a construction of H ^ given a 3D homography H . The same 
argument applies to projective camera transformations Q for 3D lines, see 
section 3. 1.2. 2. 



3 Geometric Reasoning Using Projective 
Geometry 



After we have introduced a representation of points, lines and planes within 
the framework of projective geometry, we now turn our attention to the ques- 
tion of how to perform reasoning on these entities. Geometric reasoning in- 
volves the manipulation of given entities and the derivation of new knowledge 
by checking relations between entities. 

The manipulation of geometric entities usually means to construct new en- 
tities - either from given ones, such as a line from two points or by trans- 
formation, such as the projection of an object point to an image. For the 
latter type of construction, we can use the basic projective transformation 
from the last chapter, though they have not yet been explicitly given for all 
types of transformations such as 3D lines to 2D lines, which will be worked 
out in this chapter. The construction of new entities from given ones will first 
be described by the dual operations join and intersection, for which we will 
give simple algebraic expressions using three types of matrix functions for 
these operations. The three matrix functions in turn can also be used for ex- 
pressing incidence and identity relations between the geometric entities and 
thus we gain algebraic expressions for testing relations between objects. The 
relations between objects finally enables us to express any over-constrained 
construction problem as for example any forward intersection problems in- 
volving points and lines. 

Additionally to construction and testing of geometric entities, the process 
of geometric reasoning using projective geometry involves the transformation 
between the Euclidean coordinates to homogeneous coordinates, see figure 2.2 
in the last chapter. Thus we seperate the geometric reasoning process into 
three parts, cf. fig. 3.1: 

- the transfer of Euclidean observations to a projective entity, 

- the construction and/or tests using simplified formulas from projective 
geometry and 

- the normalization of the projective entities back to a Euclidean represen- 
tation 

An important aspect here is that we don’t explicitly work with projective 
entities in P" (which is not a vector space) but with their homogeneous - 
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and thus over-parametrized - representation in the vector space This 

will allow us to use algebraic operations for vector spaces. 



transfer 




normalization 



Fig. 3.1. This diagram illustrates the process of geometric reasoning using projec- 
tive geometry: starting with a Euclidean entity x G R" we can derive a projective 
entity x G P" which we can represent as a homogeneous vector x G R"”*"^, see 
also (2.3). We may now construct a new homogeneous vector y G R’"'*'^ using 
homogeneous vectors Xi from If we are interested in the Euclidean rep- 

resentation, we apply a normalization as in (2.3), yielding y G R"*. For testing 
relations between objects, we also use the homogeneous representation instead of 
the Euclidean. 



3.1 Unique Constructions of Entities 

In this section we describe how to algebraically compute the unique con- 
struction of new entities from given ones. There are two types of unique 
constructions that are discussed in this work: the first type involves the com- 
putation using the operators join and intersection. The proposed algebraic 
expressions are equivalent to the determinant operations of the Grassmann- 
Cayley algebra, yet are solely based on a matrix-vector notation. 

The second type of unique constructions use the projective transformation 
matrices from section 2.3. Using join and intersection, we will introduce alge- 
braic expression for projective transformations involving 3D lines. Addition- 
ally, yet different transformations can be easily defined such as the inverse 
projective camera, which projects image points onto arbitrary planes. 
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3.1.1 Join and Intersection 

Constructions using join and intersection are natural ways of defining new 
points, lines and planes, see figure 3.2. The join operator takes given entities 
and extend them to the smallest projective subspace that contains the given 
entities: two (linearly independent) points form a line, three (linearly inde- 
pendent) points form a plane. The intersection operator does the opposite: 
it takes the largest projective subspace such that it is incident to the given 
entities: two 2D-lines intersect in a 2D point, three planes intersect in a 3D 
point. 

Join and intersection yield only a geometric entity if the resulting entity is 
always non-empty; for example: in general two 3D lines are not coplanar, 
in other words they do not intersect in a point and thus we don’t define 
an intersection operator with two 3D lines. Note that the resulting entity of 
a valid intersection may lie at infinity: two parallel but different 3D planes 
intersect in a line at infinity. 




Fig. 3.2. Eight unique constructions, the upper row displays joins, the lower row 
intersections. From top left to lower right: two 2D points define one 2D line; three 
3D points define a plane; two 3D points define a 3D line; a 3D point and a line 
define a plane; two 2D lines intersect in a 2D point; three planes intersect in a 
point; two planes intersect in a line; a 3D point as the intersection of a 3D line and 
a plane 



The join operator is denoted with A and the intersection or meet operator 
with n. With the linear subspaces in projective space, A and H form a so- 
called double algebra or Grassmann- Cayley algebra. In this algebra join and 
intersection but also incidence of entities are operations represented by deter- 
minants of their coordinates. We will give an equivalent description of these 
operations in terms of matrix-vector multiplication in the next chapter. For a 
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good overview of the Grassmann-Cayley algebra, see Faugeras and Luong 
2001 or Carlsson 1994. 



3. 1.1.1 Duality of Join and Intersection Implicitly we have already 
used a join operation in the last chapter, e.g. when mentioning the test on 
collinearity of three 2D points xi,X 2 ,X 2 , cf. section 2.5: they are collinear 
iff the determinant |xiX 2 X 3 | = 0, see also figure 2.8. We can infer from the 
collinearity property that the point X 3 is incident to the line 1 = xi A X 2 that 
is defined by the join of xi and X 2 : X 3 G (xi AX 2 ) Note that the 2D line may 
be expressed in Pliicker coordinates as it was done for the 3D line. The dual 
case to the collinearity of three points is the common intersection of three 2D 
lines li, 12,13 which is checked by |li,l 2 ,l 3 | = 0. As before we can construct 
a point X = li n I 2 which is the unique intersection of the first two lines and 
the property is reformulated as I 3 9 (li n I 2 ). 

From the above example it can bee seen that there exists a duality property 
between join and intersection which can be characterized by 

xAy = xny = lnm (3.1) 

or with another notation 

xAy o— • Inm (3.2) 

meaning that the join of two 2D points x A y is dual to the intersection of 
two 2D lines Inm. 

This can be extended to the 3D case with join and intersection involving 
points, planes and lines: the second row in figure 3.2 contains the dual cases 
of the first row: 



X Ay 
XAY 
XA Y AZ 
XAL 



1 n m 
AnB 
AnBnc 
AnL 



The actual existence of the duality property of join and intersection can 
be shown within the Grassmann-Cayley algebra using the Hodge-operator, 
again see Faugeras and Luong 2001 for more details. 

In the following, we will explicitly develop algebraic expressions for the join 
of two points in 2D and 3D and the join of a point and a line in 3D. From 
these two constructions, we can infer algebraic expressions for the other cases 
depicted in figure 3.2. 



3. 1.1. 2 Join of Two Points in 2D and 3D We start with the join of 
two 2D points x,y as in figure 3.2, top left. We require that the constructing 
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points must lie in the resulting line, x, y S 1. From equation (2.10) on page 24 
we know that points lie on a line if and only if x^l = 0 and y^l = 0. Thus 
the join of two points x A y is defined using the vector (or cross) product 

xAy = xxy = -yxx (3.3) 

= S(x)y = -S(y)x (3.4) 



The matrix S(x), sometimes denoted as [x]x is a skew-symmetric matrix: 



S(x) 



9x A y 

dy 



^ 0 -Xz X2 ^ 

Xz 0 —x\ 

\-X2 Xi 0 / 



(3.5) 



The matrix S(x) induces the cross-product and at the same time is the Jaco- 
bian 9x Ay/dy of the join with respect to x. The rank of S equals two. Note 
that the join is undefined for x A x = S(x)x = 0. 

The join x A y is a bilinear operation as it can be expressed linearly in x and 
y. Still, this is not a linear operation as products of two variables occur. 

The join of two Euclidean points JC, in 3D has already been introduced in 
section 2.2.3 on page 30, where the resulting 3D line consists of a homogeneous 
part Lh = Y — X and a Euclidean part Lq = X x Y . For homogeneous 
coordinates X = {Xo,XhV = (f/i, Ui, VFi, Ti)T and Y = {Yo^YhY = 
(t/ 2 , V 2 , 14^2, T 2 )^ the Euclidean points are obtained by X = XojXh and 
Y = Yo/Yh. Thus 



L=XAY= 




= XhY, 



(3.6) 

(3.7) 



fXhYo-YhXo\ 
V S(Xo)Yo ) 



The homogeneous part of L can be written in matrix- vector form and 
equation (3.8) becomes 



L = XAY 



(3.9) 



^ / Xnl -Xo\ Y 
\S(^o) 0 ) 

= TT(X)Y 



(3.10) 

(3.11) 
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TT(X) := 





( 


0 


0 


-Di 




0 


Ti 


0 


-Ui 


5XA Y 


0 


0 


Ti 


-Wi 


dY 


0 


-Wi 


Ui 


0 




Wi 


0 


-t/l 


0 






Ui 


0 


0 



(3.12) 



Thus the join X A Y can be expressed by a simple matrix-vector multipli- 
cation. The join in 3D is an antisymmetric operation as X A Y = — Y A X, 
cf. Fatjgeras and Luong 2001, p. 133, which is reflected in the matrix- 
vector multiplication: 



X A Y = TT(X)Y = -TT(Y)X = -Y A X (3.13) 

The rank of the matrix TT(X) equals 3. The join X A X is undefined as 

TT(X)X = 0. 



3. 1.1. 3 Join of Point and Line (3D) From the construction in equa- 
tion (3.12) it is possible to derive all other operations shown in figure 3.2 
on page 49. First we examine the join of a point and a line in 3D, which 
also leads to a bilinear expression, depending on the point and the line. The 
bilinear expression reveal the so-called Pliicker matrix as the third important 
matrix that we use for constructions besides S and TT. 



Linear Expression with Respeet to the Line-vector L We start from the join 
of two points in 3D, L = XAy = L = TT (X)Y, and fix the point X, which is 
emphasized by defining a matrix M := TT(X). The matrix M can be regarded 
as a linear transformation from to P®. Using the duality property for linear 
mappings from section 2.5.2, we obtain 

L = MY 0-. = A (3.14) 

Note that the dual of a point is a plane with the notation Y = A. As M is a 
matrix depending on X, we have just derived a linear expression for the join 
of a fixed point X and a line L, resulting in a plane A: 



A = X AL = TT'®(X)L 


(3.15) 


= Tf^(X)L 


(3.16) 


with TT(X) := ^ = CTT(X) = TT(X) 

oL 


(3.17) 
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where C is the dual-operator for 3D lines^, see equation (2.31) on page 44. 
Note that only bilinear scalar multiplications occur in the matrix products 
Tf^(X)L = Tr"^(X)i: of the join X A L. 



Linear Expression with Respect to the Point-vector X Up to now we assumed 
the point X to be fixed. If we now switch the roles of L and X and assume 
L to be fixed and X to be a vector variable, we are able to reformulate the 
join X A L as a matrix- vector product with a vector X and using a 4 x 4 
matrix 0 : 



A=XAL= 




-S{Lh) -L 



0(L) := 



(5X A L) 

5X 



Lo 

0 










(3.18) 












(3.19) 


/ 


0 


L3 


—L2 


-L^ 






-L 3 


0 


Li 


—L5 


(3.20) 




L2 


—Li 


0 


— Lq 




V 


u 




Lq 


0 / 





with L = {Li, L 2 , Li, LqY and X = (Xi, X 2 , X 3 , X 4 }~^ . The matrix 
0(L) has the form of a Pliicker matrix of a 3D line, cf. Hartley and Zis- 
SERMAN 2000. 

We are now able to express the join A = X A L as a bilinear matrix-vector 
multiplication 

A = XAL = Tf^(X)L = 0(L)X (3.21) 

For the remainder of this work it turns out that it is notationally advanta- 
geous to replace the matrix 0(L) by an alternative definition F(L) := — 0(L). 
The matrix T(L) is identical to the original 4 x 4-matrix, but only differs in 



the sign and the permutation of the Pliicker coordinates Ti, • • • ,Lq, see also 


section 2.2.4 on page 32. Thus (3.18) becomes: 








A = X A L = r"^(L)X 


^ 0 


Lq 


— L 5 —L^ 


(3.22) 


with r(L) := -0(L) = 0‘^(L) = 


—Lq 


0 


Li —L 2 


(3.23) 




L 5 




0 -Ls 






\ Li 


L 2 


L 3 0 / 




and f(L) := f(L) 








(3.24) 



^ Note that the dual sign in TT^(X) is a bit misleading as the dual of a linear 
mapping is its transpose. In this case, the dnal sign is a notational abbreviation 
such that we can multiply IT (X) with a line L and not with its dual L, cf. 
equation (3.28) on the following page. 
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As only the order and the sign of the Pliicker coordinates changes with respect 
to 0(L), the matrix r(L) is also a Pliicker matrix of a 3D line L, cf. Hartley 
AND ZiSSERMAN 2000. The matrix r(L) = r(L) is called the Pliicker matrix 
for the dual line or just dual Pliicker matrix^. Note that r(L) = r(L). 

The expression in equation (3.21) on the page before now becomes 

A = XAL = Tf^(X)L=f^(L)X (3.25) 

Note that equation (3.25) is written such that it resembles the fact that the 
join of a point X and a line L is a symmetrical operation in the Grassmann- 
Cayley algebra. 

Summarizing the last pages on join operations, we observe that the matri- 
ces S(-), TT(-) and r(-) play similar roles for calculation: they are the only 
matrices that are necessary to express join and intersection in matrix-vector 
notation. We call these matrices construction matrices: 

Definition 3 (Construction Matrices) The matrices S(-), TT(-) and r(-) 

defined in equations (3.5), (3.12) and (3.23) are called construction matrices 
of their geometric entities. 

The matrix S(-) is a construction matrix of a 2D point Ti or a 2D line 1. TT(-) 
is a construction matrix of a 3D point ^ or a plane A. Finally r(-) is a 
construction matrix of a 3D line L or its dual L. 

3. 1.1. 4 Dual Operations: Intersections in 2D and 3D We are now 

ready to derive the intersection operations: using the three join constructions 
in equations (3.4), (3.9) and (3.25) we can take advantage of the duality of 
join and intersection to obtain three intersection constructions: 

l = xAy =S(x)y = -S(y)x 

o— • x = lnm = S(l)m 
L = XAY = TT(X)X =-TT(Y)X 

0 -. L = A n B = TT(A)B 
A = X A L = Tf^(X)L = r^(L)X 

0 -. X = A n L = Tf'^(A)L 

With the definition 

f(L) := r(L) = r(CL) and TT(A) := CTT(A) (3.29) 

and C defined in (2.31) the intersection operators in 3D can be reformulated: 

^ In earlier work (Forstner et al. 2000, Heuel 2001, Heuel and Forstner 
2001) the definition of the matrices F(L) and F(L) is permuted. The current 
definition though provides a more consistent notation of the algebraic expres- 
sions, see table 3.1. 



= -S(m)l (3.26) 
= -TT(B)A (3.27) 

= r^(L)A (3.28) 



3.1 Unique Constructions of Entities 55 

L = A n B = TT(A)B = -Tf(B)A (3.30) 

X = A n L = Tr"^(A)L = r"^(L)A (3.31) 

It is important to note that with equations (3.30) and (3.31) we use the 
join and intersection as operations on geometric objects and not as algebraic 
operations of the Grassmann-Cayley algebra. For example, the algebraic op- 
eration of the intersection of two planes yields the dual line L, see equa- 
tion (3.27). In practice though, we are interested in the line L and not in 
its dual, therefore we “hide” the duality in the matrix-vector multiplication 
with TT(A) = CTT(A), thus we obtain a geometric operation L = A n B. 
Also note that the expression XY^ — YX^ with two points X, Y may serve 
as an alternative definition of the Pliicker matrix F as 

r(X A Y) = XY'r - YX'r (3.32) 

Equation (3.32) can be proven by comparing the matrix elements of XY^ — 
YX^ and r(TT(X)Y). Note that with two planes A,B we obtain 

f(A n B) = AB^ - BA'r (3.33) 

as f(A n B) = r(TT(A)B) = r(TT(A)B) = AB^ - BA^. 



3. 1.1. 5 Trilinear Constructions Two more unique constructions are 
shown in figure 3.2: the join of three points and dually the intersection of 
three planes. The join of three points A = X A Y A Z results in a plane A. 
Since the join operation A is associative, we may write 



(XA Y) AZ 


(3.34) 


r'^(X A Y)Z 


(3.35) 


(TT(X)Y) Z 


(3.36) 



which can be written as a trilinear expression, since 

XAYAZ = ZAXAY = YAZAX 



Dually one obtains identical expressions X = A n B n C for planes: 



(AnB)nc 


(3.37) 


r'^(AnB)c 


(3.38) 


(TT(A)B) C 


(3.39) 


f^(TT(A)B) C 


(3.40) 



and 



AnBnc=cnAnB=BncnA 



We summarize the algebraic expression for all eight unique constructions in 
table 3.1. 
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Table 3.1. Unique Gonstruction using join A and intersection n of new points 
lines and planes. The matrices S, TT, TT, F and F are defined in equations (3.5), 
(3.12), (3.17), (3.23) and (3.28). All forms are linear in the coordinates of the given 
coordinate vectors. Note that join A and intersection n are used as operations on 
geometric objects and not strictly as algebraic operations of the Grassmann-Gayley 
algebra: the duality operation in the dual line L as in L = A n B is hidden in the 
algebraic expressions, see equations (3.27)-(3.31). 



entities 


construction 


expression 


points X, y 
lines 1, m 


1 = X A y 
X = 1 n m 


l = S(x)y = -S(y)x 
X = S(l)m = — S(m)l 


points X, Y 
planes A, B 
point X, line L 
plane A, line L 


L = XA Y 
L = AnB 
A = XAL 
X = AnL 


L = TT(X)Y = -1T(Y)X 
L = Tf (A)B = -If (B)A 
A = TT^(X)L = f^(L)X 
X = TT'^(A)L = r’^(L)A 


points X, Y, Z 


A = X A Y A Z 


f'^(Tr(X)Y)Z 

= f'^crT(Y)z)x = f'^crT(z)x)Y 


planes A, B, C 


x= AnBnc 


f’^(TT(A)B)C 

= f'^(Tr(B)C)A = f'^CrT(C)A)B 



3.1.2 Transformation of Points, Lines, and Planes 



We have already seen ways to construct points using homogeneous transfor- 
mations, cf. 2.3 on page 34. Taking advantage of the duality property, it is 
easy to find the algebraic expression for transforming hyperplanes given a 
point transformation, cf. 2.5 on page 40. 

In this section we especially deal with the transformation of 3D lines given a 
3D point transformation. In particular we derive algebraic expressions for ho- 
mographies and projective cameras using join and intersection. As an example 
for an application, we will derive an algebraic expression for a transformation 
from image entities to 3D entities that are incident to a given plane. 



3. 1.2.1 Homography Given a 4 x 4 matrix H which describes a point- 
to-point homography in 3D, it is now easy to derive an expression for 3D 
lines, which is a 6 x 6 matrix Hr.: the f-th column of Hr. is the im- 
age of the canonical 6-vector ®Er = (0, • • • , 1, • • • , 0)^. With our choice of 



Pliicker coordinates, the canonical lines = ('^Ej)A("‘Efc) can be expressed 
by two canonical points, represented by the 4- vectors ^Ej and '^Ej,, where 
(j, k) = (4, 1), (4, 2), (4, 3), (2, 3), (3, 1), (1, 2) for f = 1, . . . , 6, see section 2.2.4 

(0 

on page 32. Thus we can express the f-th column H)^'^ of the line homography 
by using the canonical points '^Ej and "^Efc: 
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A HW 

= {H ■ {%)) A {H ■ i^Ek)) (3.41) 

with the choice of j, k as above. For example, assume a Euclidean rotation 
R = (ri,r 2 ,r 3 ) and a translation by X. The Euclidean motion '"H is a 
special homography and is given by 



= r ^ ’’Z t-3 x\ 

^ 0 0 1 / 

For the sake of simplicity, we only consider i = 1 and f = 4, thus (_), k) = (4, 1) 
and (2,3). All other cases are similar to the chosen two: 



‘H« = 



■H« = 



We obtain 




‘Hr, = 



R 



\S{X)R Rj 



(3.42) 



see also Navab et al. 1993. 

The computation of Hr, given a plane-to-plane homography H^i = H^ 
dually given by 



T is 






= TT(H 



Note that the derivation of the general homogeneous line-homography Hr, is 
simpler than the one of Bartoli and Sturm 2001 



3. 1.2. 2 Projective Pinhole Camera In section 2.3.2 on page 35 the trans- 
formation matrix for a projective camera P was introduced for points only, 
x' = PX. With the new constructors we are now able to construct a projec- 
tion matrix Q that can be applied to 3D lines, yielding 2D image lines, 

1' = QL (3.43) 

In the following, we will describe the derivation of the matrix Q in detail, 
since some of the previously introduced notions can be applied. 

f X\ 

Suppose we have the projective camera matrix P = I 1 , where the three 

rows A, B and C may be interpreted as three distinct planes, cf. section 2.3.2 
on page 35, intersecting in the projection center. Then the projective camera 
matrix can be derived by intersections of the three planes: 
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Proposition 1 (Projective camera matrix for lines) Given a projective 



camera matrix for points, P = 



where x' = PX, the projective camera 
( (bFtc) 



matrix for lines is given as Q(P) = Q = ( (cnA)^ ) , where 1' = QL. 

V ( aFtb)T / 



Proof: Assume a 3D line L was constructed by two 3D points X and Y, thus 
L = X A Y. Then the image line 1' of the 3D line L is given as 1' = x' A y' = 
PX A PY. Using x' A y — S(x )y , this Cctn be writt6n cis 






^B^X C^Y - C^X B^Y^ 
C^X A^Y - A^X C^Y 
^A^X B^Y - B^X A^Y j 



(3.44) 



We now have to prove that equation (3.44) is equivalent with 



r = Q(X A Y) 



( (BTTC)T (X a Y)^ 
(CtTA)T (X A Y) 
\(AnB)T (X A Y)y 



^-<XAY,BnC>^ 
-<XAY, CnA> 

\-<XAY,AnB>y 

(3.45) 



Without loss of generality we will only prove the first inner product: 



B^X C^Y - C^X B^Y = X^B^ C^Y - X^C B^Y 

= X'^(BC'r - CB'^)Y 
= X'^(r(B n C))Y 
= X'^(-Tf^(Y)) (Bnc) 

= - (Tf(Y)X)^ (BnC) 

= - (tt'^(y)x)^ (bTTc) 

= (tt'^(x)y))^ (bTTc) 

= (xay)'^b?Tc = -<XAY,BnC> 



Note that B n C is the dual of the intersection of the two planes B and C. 
Thus we could also have written B A C, meaning to take the join of the two 
points B and C. □ 

Furthermore, the dual from equation (3.43) is 

1' = QL 0-. V = QL AA Q'^F = L 

AA CQV = L 
AA Q^x' = L 



(3.46) 

(3.47) 

(3.48) 
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Here we assumed that the dual of an image line is an image point, 1' = x' and 
used the convention Q QC. Equation (3.48) reveals a constructor for a 3D 
line L given an image point x' and a projective camera, represented by Q. 
Thus the constructed 3D line is the viewing ray through the focal point, the 
image point x' and the object point X, we sometimes write the viewing ray 
as L' to stress the fact, that the line is derived by an image point: L' = Q x'. 
As in section 2.3 on page 34 we define a vector q containing the rows of the 
matrix Q 



q = vec(Q"^) 



^ bTTc ^ 
cn A 
V atTb ) 



(3.49) 



Thus the projective camera vector q for lines depends on the projective cam- 
era vector p for points and we can write q as a function q(p). One can 
compute the projective camera vector q for lines as a matrix- vector multipli- 
cation 



q = Jqp(p)p 

_ 3q(p) _ 
ap " 



' 0 -TT(C) 

TT(C) 0 
V-TT(B) TT(A) 



^B) ^ 

-TT(A) 

0 ) 



(3.50) 

(3.51) 



Note that q = Jqp(p)p is not linear in p, as entries of p occur in Jqp. 



3.1.3 Fundamental Matrix 

As we now have an algebraic expression to construct a viewing ray L' from 
an image point x' by L' = Q x', we can now give a formula for computing 
the fundamental matrix from two given cameras Pi and P 2 , see section 3.1.3: 
first, compute the line projection matrix Qi and Q 2 as described above and 
obtain the first viewing ray L' = Q^x'. Projecting L' back to the second 
image yields 

1" = Q 2 Q|x' 1" = F^^x' with F := Q 1 Q 2 = Q 1 CQ 2 (3.52) 



3.1.4 Inverse Projective Camera with Respect to a Plane 

As a demonstration of the capabilities of the described construction methods, 
we want to show a simple way to obtain the so-called inverse projective cam- 
era transformation which back-projects an image point or line onto a world 
plane (figure 3.3). This is a new transformation and can be expressed by com- 
bining a construction matrix with a projective camera matrix. The derivation 
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Fig. 3.3. Inverse projective camera: given a 
projective camera P and a 3D plane B, one 
can establish a one-to-one relation between the 
image entities x', 1' and their corresponding 3D 
entities X,L on the plane B. 



of the transformation matrix is much simpler than the one in Faugeras and 
Luong 2001, p. 185f. 

First, we demonstrate how to obtain backward projection from 2D points to 
3D points by using the join operator: assume an arbitrary plane B in object 
space is given along with a projective camera P, see figure 3.3. A point x' 
induces a viewing ray L' = Q x' because of (3.48). We can now intersect the 
viewing ray L' with the given plane B to obtain the object point X: 

X = B n L' = Tr"^(B)Q^x' = Tf^(B)Q'^x' =: P+x' 

The 4x3 projection matrix P^ can be regarded as the inverse projection 
matrix to P with respect to a plane B: for any plane B not going through 
the projection center the equation x' = PP)jx' holds. Thus the product is 
PP)j = I 3 . and the rank of P^ has to be 3. 

For image lines F, we obtain the following inverse projection matrix: 

L = B n A'(L) = Tf(B)P'^l' =: Q+F 

Summarizing all transformations given a projective camera matrix P, we 
obtain table 3.2. 



3.2 Construction Matrices and Their Interpretation 

Before we explore relations between objects in section 3.3, we take a closer 
look at the construction matrices S(-), TT(-) and r(-), defined in definition 3 on 
page 54. They have been used as linear mappings dependent on one entity and 
are relevant for algebraic expressions of join and intersection. In this section, 
we will investigate the nature of the construction matrices by interpreting 
their rows and the columns: it turns out that they can be seen as special 
geometric entities, that are incident to the entity that defines the matrix. This 
observation will help us to easily find algebraic expressions for all possible 
identity and incidence relations between entities in 2D and 3D. 
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Table 3.2. Possible constructions using a projective camera matrix P: forward 
projection (3D— >2D) and backward projection (2D^3D). The derivation of the 
last two projection matrices are given in section 3.1.4. 
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X'b = P+x' 


P+ = P+(B,p) = ^^(B)Q^(p) 


line 

line 


Vb = Q^i' 


Q+=Q+(B,p)=T(B)P^(p) 



3.2.1 Canonical Entities in Construction Matrices 

In the following we will define special points, lines and planes, that are con- 
tained in the three defined construction matrices. For every point X, line L 
or plane A, there exists so-called canonical entities that relate X, L resp. A 
with the given coordinate system in an obvious manner. We first explore the 
simpler 2D case before proceeding to 3D. 

3. 2. 1.1 Points and Lines in 2D Given a 2D point x, the construction 
matrix S(x) was used for constructions of lines passing through x using 1 = 
S(x)y. The second point y may be anywhere on the projective plane besides 
on the location of x. Choosing the three canonical points e^, see table 2.1 on 
page 28, we obtain three lines 1^, * = 1, . . . , 3, which are the columns of S(x), 
see figure 3.4(left): li is the join of x and the infinite point in the direction 
of the x-axis, thus being parallel to the x-axis and passing through point x. 
Similarly, I 2 is the line parallel to the y-axis passing through x and I 3 passes 
the origin 63 and x. Thus we can rewrite the skew-symmetric matrix as 
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^ 0 —w V ^ 




( \ 


S(x) = 


w 0 —u 


= 


ll 12 13 




\^—V U 0 y 




1 / 







^(ei A x)"'"^ 


(xAei) (xAe2) (x A 03) 


= 


(e2 A x)"^ 


V J 




V(e3 A x)Ty 



The last equality stems from the fact that the rows of S(x) are the negated 
columns, thus also correspond to the three special lines 1^. 

The same argument applies for the matrix S(l) of a line 1 = (a, b, c)^ leading 
to three special points x^ where two points xi, X2 are the intersections of the 
coordinate axes and 1 and the third point X3 lies at infinity in the direction 
of 1, thus 





^ 0 — c b ^ 




1 \ 


S(l) = 


c 0 —a 


= 


Xl X2 X3 




^—b a 0 y 




1 / 



/ ^ 




n 1)"^^ 


(inei) (in 02) (in eg) 


= 


(62 n 1)T 


V J 




V(e3 n 1) 7 





Fig. 3 . 4 . Interpretation of construc- 
tion matrix S(-) for a point x (left) 
and a line 1 (right). A point x can be 
represented by two lines li , I2 parallel 
to the coordinate axes and one line I3 
passing the origin. A line 1 can be rep- 
resented by two points xi , X2 on the 
coordinate axes and one point X3 at 
infinity. 



Considering that there is a simple bijective relation between a point x and 
its construction matrix S(x), we will consider the matrix S(x) as the matrix 
representation of the point x. 

The entities that are contained in a construction matrix are subject of the 
following definition: 

Definition 4 (Canonical entities in S(-)) The lines (e^ Ax) contained in 
the construction matrix S(x) are called the canonical lines of a point x. In 
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the same manner, we eall the points (eiHl) contained in S(l) canonical points 
of the line 1. 

Choosing a Minimum Number of Canonical Entities In the case of a point x, 
it is obvious that only two canonical lines — thus two columns resp. rows of 
S(x) — are needed to completely determine the point x. In the following we 
identify the canonical lines with the rows of S(x). As these rows are linearly 
dependent, the rank of the matrix equals the minimum number of canonical 
entities. 

Sometimes it will be necessary to choose the two linearly independent canon- 
ical lines from the matrix S(x), as we will see when doing statistical tests 
of relations, section 4.5. But the choice of the rows depends on the position 
of x: it may happen that two coordinates of x = [u, v, w)^ are zero, then a 
row of S(x) is equal to the transposed zero- vector 0 ^ and the corresponding 
canonical line (x A e^) is undefined; thus the other two rows of S(x)have to 
be chosen. 

Choosing the rows in a numerically stable manner is straight-forward: remov- 
ing the row, which is closest to 0^ yields a 2 x 3 matrix S[^l(x) := j[^lS(x), 
where is the 2x3 reduction matrix, defined by keeping only two rows 
from the identity matrix /. The closest row rj = (x A to 0^ can easily 
be chosen by comparing the norms of the rows, for example the L-1 norm 
Ikdl = l(^*)jl- Note that the reduced matrix S[^l(x) still has rank 2 , 

as the remaining rows are guaranteed to be linearly independent since the 
associated canonical lines are different. 

Of course the same argumentation holds for lines 1 and their matrices S(l), 
yielding a reduced matrix S[^l(l). 

3. 2. 1.2 Points and Planes in 3D For a point X in 3D, the construction 
matrix TT may be interpreted in two ways: TT (X) from X A Y and the dual 
transposed matrix TT (X) = TT^(X)C from XAL, see table 3.1.. The matrix 
TT (X) contains four columns, interpreted as lines L.^ = X A E.^ with the four 
canonical points E^, see table 2.2 on page 30: three lines Li , L 2 , L 3 are parallel 
to the coordinate axes and pass through the point X, the fourth line L 4 is 
incident to both the origin E 4 and the point X, see figure 3.5 (top left). 

The matrix TT (X) contains six columns, interpreted as planes = X A 
Ei with the six canonical lines E^, see table 2.3 on page 32: three planes 
Ai, A 2 , A 3 are parallel to the coordinate system planes, e.g. Ai is parallel to 
the j/z-plane and passes through X. The other three planes A 4 ,A 5 ,Ae are 
constructed by the coordinate axes and the point X, see figure 3.5 (bottom 
left) . 

Based on duality, we now know that for a plane A the matrices TT(A) and 
TT^(A) contain four lines = A n E^ resp. six points X,; = A n E^, see 
figure 3.5 (top and bottom right). 
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Fig. 3 . 5 . Interpretation of construction matrices TT for a point X (left) and for 
a plane A (right). Top left: given 1 T(X) a point X can be represented by three 
lines Li,L 2,L3 parallel to the coordinate axes and one line L4 passing the origin. 
Bottom left: given TT (X) a point X can be represented by six planes, which are 
either parallel to a coordinate system plane, e.g. A4 or intersecting a coordinate 
axis, e.g. Ai. Top right: given TT(A) a plane A can be represented by three lines 
Li,L 2,L3 incident to the coordinate system planes and one line L4 at infinity, 
incident to A. Bottom right: given TT^(A) a plane A can be represented by six 
points, three Xi, X2, X3 on the coordinate axes and the plane, three X4, X5, Xe at 
infinity (not shown). 



Analogous to definition 4 we can define canonical entities in the matrix TT(-) 
as follows: 

Definition 5 (Canonical entities in TT(-)) The points, planes and lines 
contained in TT(X), TT(X), TT^(A) and TT (A) are called the canonical 
points, planes and lines o/X and A , see table 3.3 on page 66. 

Choosing a Minimum Number of Canonical Entities As a point X is fixed 
by three parameters, it is sufficient for the matrix TT(X) to choose 3 out 
of 6 canonical planes, i.e. 3 out of 6 rows from TT(X). Again the minimum 
number of canonical entities is equal to the rank of the construction matrix. 
Similar to the reduction of the matrix S(-), we can remove the row which is 
closest to 0^, yielding a reduced matrix TT^^^(X) = J[^1TT(X) where is a 
3x6 matrix and thus (X) is a 3 x 4 matrix. 
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The rows of the matrix TT^(X) represent four canonical lines incident to 
the point X, but again three are enough to define the point. So again we 
remove one row closest to 0^, the remaining rows are guaranteed to be linearly 
independent to each other as they represent three different lines. Thus the 3 x 
6 matrix TT^^^ (X) = JI^ITT (X) contains the minimum number of canonical 
lines necessary for representing the point. 

Note that we could have selected rows of TT(X) and TT^(X) instead of 
columns, but then the reduction matrix had to be multiplied from the right. 
The reduction of the construction matrices TT(A) and TT^(A) for a plane A 
can be done in the same way as for the point matrices. 




Fig. 3 . 6 . Interpretation of construction- or Pliicker-matrix F for a line L. Left: given 
r(L), a line L can be represented by three intersections Xi,X 2 ,X 3 of the line L 
and the coordinate system planes. The fourth point X 4 is equal to the intersection 
of L with the infinite plane. Right: given r(L), a line L can be represented by three 
planes Ai, A2, A3 that are the join of the infinite points Ei and the line, the fourth 
plane A4 (not visible) is the join of the origin E4 and L. 



3. 2. 1.3 Lines in 3D Finally, for a line L and its dual L the columns of 
the Pliicker-matrices r^(L) and the F (L) can be interpreted as LAE^ resp. 
L n E^: the first three columns of r^(L) correspond to canonical points on 
the coordinate system plane, incident to the line L. The fourth column is the 
intersection of L with the plane at infinite distance. Since the Pliicker-matrix 
is skew-symmetric as S, the rows of r^(X) correspond to the same points. 

The Pliicker matrix of the dual line, F (L), contains the canonical planes, 
which are incident to the canonical points E^ and the line L itself 
Note that the matrix representations S,TT play the same role as the Pliicker 
matrix F for lines in 3D, but only S and F are usually used in Computer 
Vision and Graphics. The definition by applying canonical vectors follows 
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Table 3.3. Interpretation of rows and columns for the construction matrices of 
points, lines and planes. The skew-symmetric matrix S(-) may represent a 2D point 
X resp. a 2D line 1. The rows and columns of the matrix are entities that intersect 
or are incident to x resp. 1. The Pliicker-matrices r(L),r(L) represent 3D lines 
L, containing 4 incident points resp. 4 intersecting planes as rows and columns. 
The matrix TT(X) represents a point X, containing 6 planes or 4 lines and the 
matrix TT(A) represents a plane A containing 6 points or 4 lines, see also table 2.4 
on page 32. Note that the canonical vectors ei,Ei have different interpretations, 
depending on the context: they may represent points, lines or planes. 
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equation (3.4) of Faugeras and Luong 2001, p. 135. Again, the entities in 
r(-) are defined as follows: 

Definition 6 (Canonical entities in r(-)) The points and planes, that are 
contained in r(L) and r(L), are called canonical points and planes o/L. 

Choosing a Minimum Number of Canonical Entities As a line L is already 
defined by two canonical points from the construction matrices r(L) resp. two 
planes from r(L) we remove two rows from these matrices that are closest to 
QT, thus r[^'(L) = j[2lr(L) and f(L) = jl2ir(L). 



3.2.2 Reduction of Construction Matrices 

As seen before, we can reduce the construction matrices S(-), TT(-) and r(-) to 
a minimum number of canonical entities or rows, which still uniquely define 
the underlying entity. The minimum number is equal to the rank of the 
construction matrices. A reduction of matrices will be especially important 
when we are doing statistical tests of relations and estimation of entities, see 
chapter 4; so the reader may skip this section at the first time. 

For the reduction we delete those rows, which are closest to the zero- 

vector, thus closest to an undefined geometric entity. This way we obtain 
numerical stability for the remaining rows, which are guaranteed to be linearly 
independent. There are several criteria for closeness, e.g. we may take the Li~ 
norm. 

3. 2. 2.1 Consistent Reduction of Matrices in Bilinear Forms Now 

assume the simultaneous reduction of two matrices, such as in bilinear forms 
U(x)y = V(y)x, where the reduction for r = min(r[/,ry) is wanted (rjj 
and ry are the minimum number of canonical elements for U and V). If the 
reduction of rows in z!j^^ = L|['’l(x) and = Vl’'l(y)x is done independently 
from each other, then it is possible that yf 2 ^ 2 ^. Therefore one always 
has to delete the same rows from U(x) and V(y) to ensure equality between 
the reduced vectors and 22 . With this constraint though, it may be 
impossible to avoid zero rows 0^ in one of the matrices U(x) and V(y), e.g. 
consider S(ei) and — 8 ( 63 ), where only one common row is not 0 ^. 

But under certain conditions, the simultaneous reduction of two construction 
matrices is indeed possible. We consider six compatible pairs, that are used 
in the bilinear forms of the constructions: (S(x), S(y)) , (TT(X), TT(Y)) and 

(TT^(X), r^(L)) and their dual complements (S(l), S(m)) , (TT(A), TT(B)) 
and (TT^(A), r^(L)). We will only cover the first three as the reasoning is 
equivalent for the dual pairs. 
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For the first pair we obtain 
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with X = {xitX2,xzY y = (?/i, ?/2, 2/3)^- One can easily prove that there 
exists two common non-zero rows if and only if there exist an index i, such 
that \xiyi\ > 0 . A similar criteria exists for choosing three common non-zero 
rows from 
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There exist three common non-zero rows if and only if there exists an index 
i, such that > 0 : either the first three rows are chosen (* = 4 ) or two 

of the last three and one of the first three rows {i = 1 , 2 , 3 ). 

For the last pair of construction matrices, namely (TT^(X), F (L)), the cri- 
teria is a bit more complicated. Given 
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we can only assume that at least one Xi and one Lj are not zero entries, 
for example X = Ei, L = E4 {i = l,j = 4 ). Checking all combinations of 
(i, j), only for the following pairs of (f, j), one can guarantee that at least two 
common rows are non-zero: 



line index j 





1 


2 


3 


4 


5 


6 


1 


• 








• 


• 


point 2 




• 




• 




• 


index i 3 






• 


• 


• 




4 


• 


• 


• 









The table above reminds of the pattern of the non-zero entries in the matrix 
TT(X) and we can formulate the condition for consistent reduction as follows: 
if there exists a pair of indices (i,j) such that |TrJ^j (X) Lj \ > 0 , a consistent 
reduction of both construction matrices is possible. Here, Tr^j(X) denotes 
the entry of the matrix TT^(X). 

Summarizing the criteria for a possible consistent reduction of the construc- 
tion matrices, we obtain 
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Consistent reduction of construction matrices 

Objective-. Given two objects x and y (points, lines or planes in 
2D and 3D) and the construction matrices U(x) and V(y) with 
ru and rv being the minimum number of canonical entities for U 
and V. 

Return consistently reduced matrices U^’^^(x) and V^’’^(y) with 
r = min{ru , rv) ■ 

1. If the requirements in equation (3.53)-(3.55) are not fulfilled, 
rotate and possibly translate the coordinate system accord- 
ingly. 

2. Choose r rows uj, vj from U(x) and V(y) with the largest 
entries |ui||vi|. One obtains U^’’'(x) and Vi’''(y) 



Algorithm 3.1: Simultaneous and consistent reduction of pairs of construction 
matrices. It is guaranteed that the reduced matrices Lli’'i(x) and Vi’'i(y) have rank r. 



(x,y): 


3 * : \xiVi\ > 0 


with x, y being points or hyperplanes 

(3.53) 


(X,L) : 


3i,j: |TrT^(X)L,| >0 


with 3D line L and point X 


(3.54) 


(A,L) : 


3 ■ \TtJj{A)Lj\ > 0 


with 3D line L and plane A 


(3.55) 



Thus only in very rare cases a reduction of construction matrices is not pos- 
sible, e.g. for the 3D plane A = Ei and the line L = E 3 . When checking 
geometric relations, see section 3.3.4 on page 79, these cases practically do 
not occur between entities. 

If we still want to ensure that a consistent reduction is always possible, it is 
important to note that the consistent reduction depends on the choice of the 
coordinate system for the geometric entities. Therefore we may transform 
the coordinate system, e.g. by a rotation and a translation, such that the 
conditions in (3.53) - (3.55) are fulfilled. 

Algorithm 3.1 shows how to reduce construction matrices for all possible 
cases discussed above. 



3.2.3 Nullspaces of Construction Matrices 

Nullspaces of construction matrices are of interest when introducing statisti- 
cal notion to geometric reasoning, see chapter 4. One may skip this section 
for the current chapter. 

The determination of the nullspace of a construction matrix can be accom- 
plished using the interpretation of the matrix rows: for the ’Tine-from-point” 
matrices S(x) and TT(X), the nullspace is equal to the point x resp. X, as 
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this is the only point that is incident to the hyperplanes in the rows of S(x) 
resp. TT(X). Similarly for the dual cases S(l) and TT(A), the nullspaces are 
determined by the homogeneous vectors. 



Table 3.4. The two tables show the nullspaces of construction matrices for points 
and lines (left) and for planes and dual lines (right). The operator T{-) denotes the 
image or column-space of a matrix, see text for details. 
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For TT^(X), we are looking for all 3D lines L^, that are incident to the point 
X, as then TT^(A)Li = 0. But exactly these lines are contained in the 
column-space or image X(TT(X)), thus Tr^(X)TT(X) = 0. Note that the 
dimension of the column-space I(TT(X)) of TT(X) is 3, although only two 
degrees of freedom are possible for a 3D-line incident to X: the additional 
rank is due to the homogeneous scale. 

Similarly one obtains the nullspace for the Pliicker matrix: there, the column- 
space of its dual is its nullspace. All nullspaces are summarized in table 3.4. 

Summary In section 3.2 we have shown how to interpret the rows and columns 
of a construction matrix of a geometric entity. A summary of the interpreta- 
tion of all construction matrices is shown in table 3.3 on page 66. Thus any 
matrix (or linear map), which has the form of one of the homogeneous matri- 
ces S(-), TT(.), TT(.), r(.) or f(.) can be interpreted as an over-parametrized 
representation for a geometric entity. One may reduce the number of rows in 
the matrices but still keep a complete representation for an entity. 



3.3 Relations between Entities 

In this section we investigate geometric relationships between entities con- 
structed in the previous section, concentrating on pairwise relations. Espe- 
cially we will cover four kinds of relations: incidence, identity, orthogonality 
and parallelity. Furthermore we provide simple computations for the Eu- 
clidean distances between various entities, such as the distance between a 3D 
line and a point. 
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3.3.1 Projective Relations 

The first type of relations that we explore are projective relations, i.e. rela- 
tions that are invariant with respect to a projective transformation. Examples 
of these relations are collinearity, coplanarity, incidence and identity. 

3. 3. 1.1 Collinearity and Coplanarity In section 2.1.2 on page 23 we 
have found that we can use a determinant form to check the collinearity 
resp. coplanarity of three points in 2D resp. four points in 3D: 

xi,X 2 and X 3 are collinear | xi X 2 X 3 | = 0 (3.56) 

Xi,X 2 ,X 3 and X 4 are coplanar | Xi X 2 X 3 X 4 | = 0 (3.57) 

Taking the dual equations from (3.56) and (3.57) one obtains for 2D lines b 
and planes A.^: 

li, I 2 and I 3 intersect in one point | li I 2 I 3 | = 0 (3.58) 

Ai, A 2 , A 3 and A 4 intersect in one point | Ai A 2 A 3 A 4 | = 0 (3.59) 

see also figure 2.8 on page 41. 

To check collinearity, coplanarity or intersection, we need at least 3 entities. 
For the remainder of this work we concentrate on checking pairwise relations, 
for example the incidence of a point and a line in 3D. Checking collinearity be- 
tween three points can then be accomplished by first constructing a line from 
two points and check the incidence of the third point with the constructed 
line. 

3. 3. 1.2 Incidence of Dual Entities Equations (3.56)- (3.59) show how 
to check some geometric relations using the determinant form. As in equa- 
tion (2.27) on page 42 we may reinterpret the entities involved in the deter- 
minant: by constructing a plane A = Xi AX 2 AX 3 from the first three points 
and check whether the fourth point X 4 is incident to the plane: 

Xi,X 2 ,X 3 and X 4 are coplanar | Xi X 2 X 3 X 4 | = 0 

AA A^^X =0 

<S4> < A,X > =0 

X 4 e A 

with A = (Xi AX 2 AX 3 ). For lines, we can derive a similar expression using 
the cap-product < L,M > as already see in section 2.5.1: 

Xi,X 2 ,X 3 and X 4 are coplanar AA | Xi X 2 X 3 X 4 | 

4A L'rM 
AA < L, M > 

AA LnMy^0 



= 0 
= 0 
= 0 
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with L = Xi A X 2 and M = X 3 A X 4 , see also equations (2.28) and (2.29) 
on page 42. 

Both incidence relations are checked by using the cap-product < • >: we 

can express the following incidence relations for dual entities using the cap- 
product: X G 1, X G A and L n M yf 0: 



d = 


(x, 1 ) = x^l = 0 


(3.60) 


d = 


(X,A) = X^^A = 0 


(3.61) 


d = 


(L, M) = = 0 


(3.62) 



To check whether the incidence relation holds, one has to check whether 
d= 0. 

There exists no cap-product for the other projective relations, for example 
the incidence of a 3D line and a plane. For these relations, we have to find 
the corresponding algebraic expressions, which is the goal of the following 
paragraphs. 

Remark For the incidence relation L n M yf 0 AA L^M = 0 we can give 
another proof based in the algebraic expressions for constructions: assume 
two lines L = X A Y and M = Z A W. The two lines are incident if and only 
if the four points are coplanar. Thus we construct a plane 

A = XAZAW = XAM = TT^(X)M = Tr"^(X)M 

We can now write: 

Y G A Y'r (TT'^(X)M) = 0 (TT(X)Y)'^M = 0 L'^M = 0 

3. 3. 1.3 Identity of 2D Points x = y and 2D Lines 1 = m Checking 
the identity of two homogeneous vectors from is similar to checking the 
collinearity of two directional vectors r, v G in 3D: they are identical if 
and only if their cross product is zero, i.e. collinear(r, t>) AA r x = 0. The 
geometric interpretation is obvious as the cross-product yields the vector 
orthogonal to r and v and is undefined if they are collinear. 

Thus the identity of points in 2D can be checked by 

d = S(x)y = 0 (3.63) 

Using the interpretation of S(x) from section 3.2, the equation S(x)y = 0 
performs three incidence tests of the point y with the three lines x A , i = 
1,2,3. In general only two incidence tests are sufficient to determine the 
validity of the relation, as the test has two degrees of freedom. Thus we can 
use the reduced matrix S[^l(x), mentioned on page on page 63 and which 
contains two canonical lines. Thus the check for identity can be reformulated 



as 
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d(2] = s[2] (x)y = _S[21 (y)x = 0 (3.64) 

The identity of 2D lines 1 = m is checked in exactly the same manner. 

The reduction is computed as in algorithm 3.1. Note that if one does not 
apply the proposed coordinate transformation in step 1 of the algorithm, the 
reduction fails only for cases, where the two vectors contain zeroes and are 
exactly orthogonal, V i : \xiyi\ = 0, see also equation (3.53). If one knows 
that the lines are close to being identical, thus very far from being orthogonal, 
the rotation step may be skipped. 

3. 3. 1.4 Incidence L 6 A of a 3D Line L and a Plane A For the relation 
of a 3D line and a plane we can use a construction matrix as in the check 
for the identity of 2D points and lines: the line L has four canonical points, 
represented as columns of the Pliicker matrix F^(L), see figure 3.7. In general 
it is sufficient to check whether two canonical points from F^(L) are incident 
to the plane A, so the test has 2 degrees of freedom. The test may also 




Fig. 3.7. Checking the incidence of a 3D line L with 
a plane A by checking the incidence of the canonical 
points of L with the plane A. 



be performed by choosing two columns of TT(A) as they correspond to two 
canonical lines of the plane A. The canonical lines may then be checked for 
incidence to the given line L, by TT(A)L = TT(A)L, see also equation (3.62). 
So the algebraic expression of the test is bilinear: 

D = r(L)A = TT'^(A)L = 0 (3.65) 

We may use the reduced construction matrix (L) cf. page 67 to test the 
incidence of the plane with only two out of four canonical points. Likewise, 
we choose two rows from TT^(A) yielding TTf^^^(A) according to the algo- 
rithm 3.1 on page 69. 

D(2] = r[^l(L)A = TT[21'^(A)L = 0 (3.66) 

Again, the proposed transformation in step 1 of the reduction algorithm is 
only needed for special cases of lines and planes: V i,j : |TTjj(A) Lj\ = 0. 
Such cases happen only if the incidence relation is not fulfilled at all: for 
example, if A is the infinite plane and the line L is incident to the origin. In 
general, this may only happen if A G L. 
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3. 3. 1.5 Incidence X 6 L of a 3D Point X and a Line L This is the dual 
case of the previous test and thus we obtain the bilinear algebraic expression 

D = f(L)X = Tf^(X)L = 0 

and 

= f'^'(L)X = Tt'^''^(X)L = 0 

If one skips the coordinate transformation step in algorithm 3.1, then a re- 
duction may be not possible for special cases and if X G L instead of X G L, 
see equation (3.54) on page 69 for an exact condition. 

3. 3. 1.6 Identity of 3D Points X = Y and Planes A = B Since there 
is no simple test for the collinearity of directional vectors in we adopt the 
above method of using the construction matrices: if the point Y is incident 
to the six canonical planes contained in the matrix TT(X), then the points X 
and Y are identical: 



D = TT(X)Y = -TT(Y)X = 0 

The test has 3 degrees of freedom, thus it is sufficient to check three incidences 
as long as the canonical planes are defined, i.e. non-zero. Thus we may write 

D(3] = tt[^1(X)Y = -TT[^'(Y)X = 0 

The reduction of the construction matrices is valid as long as the points are 
not at infinity. 

The identity of 3D planes A = B is the dual case to the identity of points 
and thus 

D = Tf(A)B = -Tf(B)A = 0 

and 

D[3] = Tf'^'(A)B = -Tf'^'(B)A = 0 

For the reduction of the matrices, we apply the algorithm 3.1 again. As in 
the case of identity of two lines, the proposed coordinate transformation in 
step 1 of the algorithm is only needed for special cases, where two planes 
are exactly orthogonal and only one coordinate of each plane is non-zero, cf. 
equation (3.53) on page 69 

3. 3. 1.7 Identity of 3D Lines L = M The expression (L,M) = 0 only 
checks if two 3D lines share one common point. To check the identity of all 
points on the lines, we use the dual Pliicker-matrix T(L) for line L and the 
Pliicker-matrix r(M) for line M: the matrix product 

D = r(L)r(M) = 0 



(3.67) 
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contains 16 cap-products of the canonical planes of L with the canonical 
points of M. In general 4 incidences are sufficient for this test, as long as the 
corresponding columns and rows are non-zero. Applying a reduction to both 
matrices yield 

D[2.2] = f'^l(L)r[^l(M) 

The matrix is a 2 x 2 matrix containing 4 different incidence tests of 

canonical points of M with canonical planes of L, thus we have the desired 
number of tests. We may choose those rows of r(L) resp. columns of r(M), 
which have the largest norm. 

The difference of the 3D-line-identity relation in contrast to the previous ones 
is the fact that the distance test in equation (3.67) is a matrix-matrix product. 
To turn it into a matrix- vector product of the bilinear form D = vec(D) = 
U(L)M = V(M)L, we can take advantage of the Kronecker product: using 



vec(r(L)) = JpCL with 



( 0 0 0000 -1 0 0 100 0 0 0 0\ 
0 0 1000 0 0 -1 000 0 0 oo \ 
0 -1 0010 0 0 0 000 0 0 oo| 
000100000000 -1 0 ool 
0000000100000 -1 00 / 
0000000000010 0 -1 0 / 



the bilinear forms for vec(D) depending on L and M are 
D = vec(D) = vec(f(L)r(M)) = (r'^(M) 0 k) vec(f(L)) 

= (r'^(M) 0 / 4 )Jr C 1 L = 

= (/4 0r(L)) vec(r(M)) 

= [(/4 0f(L))Jr] M 





(3.68) 




(3.69) 


V(M) L 


(3.70) 




(3.71) 


U(L) M 


(3.72) 



The matrices U(L) and V(M) are of dimension 16 x 6 and must be at least 
of rank 4. 

The rows of U(L) resp. V(L) can be regarded as dual representations of special 
3D lines: using the interpretation of the rows and columns of the Pliicker 
matrices (table 3.3), the (*, j)-th element of the matrix D = r(L)r(M) is 



(LAE,, -(MnE,))) 


(3.73) 


(^(E,)L)^TT(E,)M 


(3.74) 


LT (^^(E,)TT(E,)) M 


(3.75) 


(-rr(E,)^^(E,)) L 


(3.76) 



As hij := Tr(Ei)TT(Ej)^L is a valid 3D line, the matrix element Dy can 
be interpreted as a cap product (L^ , M) between two 3D lines, and likewise 
(M,„ L). 

Unfortunately, it is not easy to see how to simultaneously choose r = 4 rows 
of U(L) and V(M). In the following section we will describe an easier way to 
obtain UM(L) and VM(M). 
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3. 3. 1.8 Yet Another Algebraic Expression for Identity Relations 

We finally describe a different way to check the equivalence of homogeneous 
vectors. We will see that it effectively leads to the same bilinear forms as 
proposed above, but can be more efficient to compute. We use the equiva- 
lence L = M of lines as an example, the same can be done for points and 
hyperplanes in 2D and 3D. 

The method is based on the proportionality of the homogeneous coordinates: 
given two lines L and M, we choose an index i = 1, • • • 6 such that \LiMi\ > 0. 
We then scale both vectors such that for both the i-th coordinate is LiMi. 
The difference T>A.i of the scaled vectors must be zero: 

Dziy = M,L - LM = 0 (3.77) 

We can write this in a bilinear form 

Dzi,i = Ai(L)M = -A,(M)L with A,(L) := LEJ - L./g (3.78) 

Especially compared to equation (3.73), we gain computational efficiency at 
the expense of choosing an index i. One advantage here is though, that the 
matrices Ai(L) and Z\i(M) share four linearly independent rows: e.g. consider 
Ai(h) for j = 4, 



l~Li 


0 


0 


Li 


0 


0 1 


^ not a valid 3D line 


0 


-L4 


0 


L2 


0 


0 




0 


0 


-L4 


L3 


0 


0 




0 


0 


0 


0 


0 


0 


^ not a valid 3D line 


0 


0 


0 


^5 


-Li 


0 




V 0 


0 


0 




0 


-lJ 





In general, only the first and fourth row do not correspond to a valid 3D 
line - the first one due to the Pliicker condition. Skipping both of these rows 
leaves a reduced 4x6 matrix Z\ 4 '^(L) with rank r = 4. The resulting rows 
of Z\ 4 ’'^(L) are contained in the 16 x 6 matrices U(L) and V(M): the 6x6 
matrices = Tr^(Ei)Tr(Ej), i ^ j are antisymmetric matrices with only 
two non-zero entries, -1-1 at position (s,t) and -1 at position (t,s) or vice 
versa. Only the pairs (s,t) = (1,4), (2,5) and (3,5) do not occur due to the 
Pliicker condition. 

Thus the matrices Z\[’^^(L) and Z\[’'^(U) can be regarded as a (not necessarily 
order and sign-preserving) reduction of U(L) and V(M). 

Note that the reduction only holds if there exists an i such that \LiMi\ > 
0. But if LiMi = 0 for all i, the reduction may fail. Using a coordinate 
transformation step as in algorithm 3.1, one can ensure a reduction for all 
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X 



tf> ■ 





Fig. 3.8. Visualization of incidence (right) and identity relations (left) for 2D and 
3D entities. The algebraic expressions for checking these relations can be found in 
table 3.5. 



possible lines L and M. In this case, only a rotation is needed, e.g. a random, 
non-identical rotation would do. 

To summarize the results of this section, all projective relations together with 
their algebraic expressions are listed in table 3.5 on the next page, see also 
figure 3.8. 

3.3.2 AfRne and Similarity Relations 

The parallelity relation between lines and/or planes is an affine relation, i.e. 
it is invariant under affine transformations, but not under projective trans- 
formations. This observation is easy to prove since the intersection of parallel 
lines and/or planes is at infinity; the 2D line at infinity 03 respectively the 
3D plane at infinity E4 remains invariant under affine transformation, cf. 
equation (2.21) on page 35. 

With the proposed representation for lines and planes the check for parallelity 
is simple: The homogeneous parts It of a 2D line, Lh of a 3D line and Ah of a 
plane contain the directional information, either as a normal for hyperplanes 
or as the direction vector for the 3D line, thus collinearity of two lines or two 
planes in 3D can be checked using the above mentioned collinearity relation 
of directional vectors r, v: 

collinear(r, v) r x v = 0 d= S(r)v = 0 

A plane and a line are parallel if the normal of the plane is orthogonal to the 

-r I 

direction of the line, thus a simple dot-product d = LhAh = 0 is sufficient. 
For two 2D lines 1 and m, we compute the orthogonal direction = (b, — a)^ 
of a homogeneous part lh = (a, b)~^ and check if it is perpendicular to m,h by 
{liVrUh = 0 . 

Orthogonality in general is not an affine transformation but is invariant under 
similarity transformations. The tests for lines and planes in 2D and 3D use 
the same methods as above and are listed in table 3.5, see also figure 3.9. 



78 



3 Geometric Reasoning Using Projective Geometry 



Table 3.5. Possible pairwise relations between points, lines and planes in 2D and 
3D. The matrices S(-), TT(-), TT(-), r(-) and r(-) are defined in equations (3.5), 
(3.12), (3.17), (3.23) and (3.28). The matrix Zli(L) is defined in (3.78), where the 
index i is chosen such that \LiMi\ 0. The degrees of freedom (dof) are valid for 
all unconstrained entities. 



type 


dim 


entities 


notation 


algebraic expression 


dof 


projective 


Incidence 


2D 


point X, line 1 


X e 1 


x"^! = I'^x = 0 


1 


3D 


point X, plane A 


X G A 


X’^A = A’^X = 0 


1 


point X, line L 


X e L 


Tr'^(X)L = f^(L)X = 0 


2 


line L, line M 


LnM/0 


L^M = M^L = 0 


1 


line L, plane A 


L G A 


TT(A)'^L = r'^(L)A = 0 


2 


Identity 


2D 


points X, y 


X = y 


S(x)y = -S(y)x = 0 


2 


line 1, m 


lEE m 


S(l)m = — S(l)m = 0 


2 


3D 


points X, Y 


X = Y 


■fT(X)Y = -■fT(Y)X = 0 


3 


lines L, M 


L = M 


f(L)r(M) = r(M)f(L) = 0 
Ai(L)M = A(M)L = 0 


4 


planes A, B 


> 

III 

CD 


TT(A)B = -Tf(B)A = 0 


3 


affine 


Parallelity 


2D 


lines 1, m 


1 II m 


{liYmh = irniYlh = 0 


1 


3D 


lines L, M 


L II M 


S{Lh)Mh = -S{Mh)Lh = 0 


2 


planes A, B 


A II B 


S{Ah)Bh = —S{Bh)Ah — 0 


2 


line L, plane A 


L II A 


Lh-^h AfiLh 0 


1 


similar 


Ortho- 

gonality 


2D 


lines 1, m 


lAy 


llruh = mllh = 0 


1 


3D 


lines L, M 


LTM 


L\Mh = MILh = 0 


1 


planes A, B 


ATB 


A\Bh = B\Ah = 0 


1 


line L, plane A 


LTA 


S{Lh)Ah = —S{Ah)Lh — 0 


2 


Euclidean (metric) 


Distance 


2D 


points X, y 


d(x,y)=s 


\xhVo-yhXo\ 

IXhVhl 


1 


point X, line 1 


d(x, l)=s 




|x'l| 

^h^h\ 


1 


3D 


points X, Y 


d(X,Y)=s 




Xhi o — h/iAo| 
|X,Y,| 


1 


lines L, M 


d(L,M)=s 




|l'm| 

S(Lh)Mh\ ' 


1 


point X line L 


d(X,L)=s 




i){Lh)X o + XhLo\ 
\XM 


1 


point X, plane A 


d(X,A)=s 


|X'A| _ 

\XhA^\ 


1 



3.3 Relations between Entities 



79 



3.3.3 Distance Relations 

The last type of relation we describe in this work are the Euclidean distance 
relations between two entities. For example the distance between a point X 
and a line L in 3D is assumed to be s, thus we check whether d(X,L) = s. 
A list of all possible distance relations can be found in table 3.5, we will skip 
a detailed proof in this work and only sketch the main ideas. 

For all relations including a point X, one can apply a translation T, such that 
the point Tx is equal to the origin. This transformation is also applied to the 
other entity, say L' = T^L for lines, where Tj;, is from equation (3.42). Then 

d(X, L) = formula for the distance between two lines L, M can be 

followed from the proof that two lines are incident, see page 43 and Brand 
1966, p. 54. 




Fig. 3.9. Visualization of par- 
allelity (top) and orthogonality 
relations {bottom) for 2D and 
3D entities. The algebraic expres- 
sions for checking these relations 
can be found in table 3.5. 



3.3.4 Checking Geometric Relations 

Given the relations in table 3.5, we can formulate a generic algorithm for 
checking projective, affine and similarity relations R using the bilinear form 

^(x,y) d = U(x)y = V(y)x = 0 (3.79) 

It is sufficient to check the r largest absolute values of the distance vector d, 
where r is the degree of freedom of the relation i?(x,y). 

We may also use reduced matrices U(x)[’'l and V(y)t’'] obtained by the algo- 
rithm 3.1, yielding 

i?(x,y) dM = UM(x)y = VM(y)x = 0 (3.80) 

Note that the coordinate transformation step of the reduction algorithm 
is only needed for special cases, cf. equations (3.53), (3.54) and (3.55) on 
page 69, where the relation i?(x,y) is not fulfilled at all. As we have seen 
in section 3.3.1, these cases do usually not occur in practice. The reduced 
distance dt’’! will become handy when doing statistical tests, see chapter 4. 
The simple method for check relations in 2D and 3D is summarized in algo- 
rithm 3.2. 
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Checking geometric relations 

Objective-. Given two geometric entities x and y (possibly in 2D 

and 3D) and a relation R € {equal, incident, orthogonal, parallel}. 

Check whether the relation i?(x, y) holds, given some tolerance e. 

1. Obtain Jacobian U(x) (or alternatively) V(y) according to ta- 
ble 3.5, such that d = U(x)y (or V(y)x). 

2. Compute d = U(x)y and check whether di < e for all coor- 
dinate di of d. It is sufficient to check the first r largest di, 
where r is equal to the degrees of freedom of the relation, see 
last column of table 3.5. 

2’. Alternative: one can compute the reduced distance vector 
_ uf’'l(x)y using the reduction algorithm 3.1 on page 69. 



Algorithm 3.2: A non-statistical algorithm for checking geometric relations. The 
alternative of the second step is computationally more involving, but is useful for 
chapter 4. The choice of a small number e « 0 in general is data-dependent and 
thus can not be globally fixed. For practical applications it is advisable to use the 
statistical version of this algorithm, see table 4.4 on page 136. 



3. 3.4.1 Checking Relations with Euclidean Distances Note that for 
some of the projective relations we can directly replace the above methods 
by checking whether the Euclidean distance d(x, y) < e is sufficiently close to 
zero. Checking for Euclidean distances has the advantage that the threshold 
e can be interpreted with respect to the application. This is available only 
for 6 out of the 10 projective relations, see section 3.3.3 on the page before 
Another option is to take advantage of the fact that for projective relations 
the distance vector d = U(x)y only contains incidence relations between dual 
entities. Thus we interpret each row of the construction matrix U(x) as an en- 
tity zj - the canonical entity - and compute the Euclidean distances d{zi,y) 
between y and z^ as above. The Euclidean distances can be interpreted and 
checked against a given Euclidean distance threshold e. Note that these dis- 
tances depend on the choice of the coordinate system as the canonical entities 
of a construction matrix are defined by the coordinate system. We don’t de- 
scribe this approach in detail as we don’t recommend the algorithm 3.2 or any 
variants that directly compute Euclidean distances: for practical applications 
one should use the statistical algorithm introduced in section 4.5. 



3.4 General Construction of Entities 

The construction methods in section 3.1 always result in a unique geometric 
entity, which may lie at infinity. But Computer Vision and Photogramme- 
try tasks often involve non-unique construction of entities: a simple example 
is the classical forward intersection of image points given two cameras, see 
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figure 3.10(a): the entity point X is supposed to be fixed by two viewing 
rays L',L", but in general these rays do not intersect exactly, due to noise 
and model errors. Thus we have to find a compromise, that fits best to the 
viewing rays. A more complex example is the construction of a 3D line given 
point and image observations, see figure 3.10(b). 

There are lots of other examples of constructing geometric entities, including 
but not limited to the following tasks: 

- corner construction: construct a junction point from a set of intersecting 
image line segments 

- grouping: construct a line from a set of collinear image points and image 
line segments 

- 3D- grouping: construct a polyhedral line given incident, parallel or orthog- 
onal lines and planes 

In this section we describe a direct method to construct the unknown entity 
for the above and many other constructions. Note that we do not take ex- 
plicitly the uncertainty of the involved entities into account. Therefore the 
solutions are sub-optimal, for a treatment with uncertainties see chapter 4. 



Fig. 3.10. Examples of non-unique construction of geometric entities: forward 
intersection with two viewing rays resulting a point X (a) and forward with two 
viewing rays and two viewing planes, resulting in a 3D line L„ (b). 



3.4.1 Relations as Constraints 

In the example of figure 3.10(a), the ideal situation is that the two rays 
intersect, L' n L" 0, so that the point X is well-defined. If this is the case, 
the point X is incident to both rays X G L',L". If this is not the case, we 
require that the point should be as close as possible to the rays, such that it 
is “almost” incident. 




2 , 



3 
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We can now reformulate the construction task as a minimization problem, 
where we want to minimize the contradiction w(-), that the variable point X 
is incident to both rays: 

w({X e L',X e L"}) ^ min (3.81) 



A point X which minimizes the contradiction w({X € L', X € L"}) is defined 
to be the best fit to the construction problem. We can now take advantage of 
the previous sections and directly give an algebraic expression for the above 
contradiction: 



w({XeL',XeL"}) 



(r\v) 

Vr^L") 



X 



(3.82) 



since X e L T^(L)X = 0. 

For the construction in figure 3.10(b), the fitted line minimizes the con- 
tradiction 



w({L„€A;, L,nL'2^0, L, ga( 5, L,nL'4^0}) = 



(L2V 

TT(A') 

V (L'4)^ / 

(3.83) 

In general we may link an unknown geometric entity x with a set of observed 
entities by any relation given in section 3.3 on page 70: incidence, identity, 
parallelity and orthogonality. This always yields to a contradiction of the form 
Mx = 0. In table 3.6, the possible relations for unknowns and observations 
are listed. A new observation can easily plugged into the set of contradictions 
by adding the appropriate construction matrix to the matrix M. 



Table 3.6. Possible relations between a list of observed entities and unknown 
entities. In 2D, we may observed points yi and lines uii and want to estimate an 
unknown point x or an unknown line 1. In 3D, we may observe space points Yi, 
lines Mi, and planes Bi and want to estimate an unknown space point X, line L 
or plane A. 



3D 


obs.^ 
1 unk. 


Y. 


Mi 


Bi 




X 


= 


G 


G 


/3 


L 


G 


= , II, -L 


e, II, -L 




A 


G 


e, II, -L 


II, -L 



2D 


obs.— > 
1 unk. 


Yi 


nii 




X 


= 


G 


1 


G 


II, -L 
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In general, we have n relations Ri{yj,(3), where yj are Wi observations that 
are related to the unknown f3. In 3D, the maximum number n of different 
relations is 3 for unknown points, 8 for unknown lines and 7 for unknown 
planes, cf. the rows in table 3.6. 

We obtain the following algebraic expression between the unknown and the 
observations: 



w ({i?i (yi , /3) , • • • , i?i (yu,i , /3) , • • • i?„(yu,„ , /3)}) 



'^Ui(yi)^ 

\Un(yn)/ 



/3 = A/3 =0 



(3.84) 

Alternatively, if we assume that the entities are not at infinity, we may also use 
the reduced construction matrices LJ[’'‘l(yj), where is the degree of freedom 
of relation R{yj,f3), cf. last column in table 3.5. Note that we assume that the 
relation R{yj,(3) approximately holds, thus we do not have outliers such as an 
observed line that is supposed to be identical to the unknown, but actually is 
orthogonal. Therefore we can safely skip the coordinate transformation step 
in the reduction algorithm 3.1. 

The modified minimization problem is now 



w' ({i?i(yi,/3),- • • ,i?i(y„,i,/3), - • •i?„(y„,„,,3)}) = 



\uL''”'(y„)/ 



/3 = A',3 =0 



(3.85) 

Assuming the reduced matrix A' is an n x u matrix, the problem is generally 
solvable, if n > dof(/3), where dof(/3) denotes the degree of freedoms of the 
unknown entity (3. Hence the redundancy is given by red = n — dof(/3). To 
compute the redundancy for a specific 3D problem, check table 3.7. 

The proposed formulation of the geometric estimation problem contains a 
large variety of combinations between an unknown entity and related ob- 
served entities. 



3.4.2 Minimizing Algebraic Distance 

To derive a solution to the minimization problem, we first reformulate it: 
instead of stacking the construction matrices in a large matrix A as in the 
examples of the previous section, we may also minimize the squared sum 

I? = = /3^ I ^ UjUi I /3 ^ min (3.86) 

Since the unknown vector /3 is a homogeneous vector, we need an additional 
constraint to obtain a unique solution to the minimization problem, for ex- 
ample by fixing its length using /3^/3 = 1. 
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Table 3.7. Galculating redundancy for geometric estimation problems: the cells of 
this table show the number of effective observations when estimating an unknown 
3D entity using equation (3.84) resp. (3.85). For example, consider the forward 
intersection in figure 3.10(b): the unknown line L has um = 2 incident lines and 
ub = 2 incident planes. Then the redundancy is given as red = nM + 2ns — dof(L) = 
6-4 = 2. 



unknown 

entity 


observations 


€ 


= 


II 


T 


ny points 
riM lines 
riB planes 


my points 
mjvr lines 
ms planes 


Pm lines 
Pb planes 


qM lines 
5s planes 


point X 
(3 dof) 


2 um + Ub 


3my 


- 


- 


line L 
(4 dof) 


2ny + Um + 2riB 


4mM 


2pM -f Pb 


QM + 2gs 


plane A 
(3 dof) 


ny + 2 um 


3m B 


Pm + 2.pB 


2qm + Qb 



The solution (3 to the problem 17 — *■ min with /3^/3 = 1 is given by the 
eigenvector to the smallest eigenvalue of U = UiUj The solution is known 
to be the total least squares estimate cf. Duda and Hart 1973, p. 332f., 
377f. 

Practically, it is advisable to compute the total least squares solution using a 
singular value decomposition, Press et al. 1992: with U from above, compute 

U = EDG"^ (3.87) 

where E and G are orthogonal matrices and D is a diagonal matrix with non- 
negative entries, that are sorted in descending order. Then the estimate (3 is 
the last column of G. Alternatively one can use A' from (3.85) instead of U. 



3.4.3 Enforcing Pliicker Constraint 

When computing the linear solution (3 from the last section, only the homo- 
geneous constraint was taken into account. In case of an unknown 3D line 
there is the additional Pliicker constraint (2.20), which may not be fulfilled 
by the estimated L. This happens, when the observed entities are subject 
to small noise. In this case we need to enforce the additional constraint to 
update the estimation to a new L. 

As suggested by Kanatani 1996 and Matei and Meer 2000 it is possible 
to use an orthogonal projection onto the manifold of all valid 3D lines, which 
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is defined as the set of all homogeneous 6-vectors, that fulfill the Pliicker 
constraint p(L) = L L = 0. We can approximate the manifold locally by 
ap(L)/9L = 

L = L-iy^E (3.88) 

2 |LP 

Since this is only a local approximation and the constraint is non-linear, one 
has to iterate a couple of times until the Pliicker condition is met. 

To avoid the iteration process, one can enforce the constraint for the Eu- 
clidean and homogeneous part separately: one can either force the homoge- 
neous part to be orthogonal to the Euclidean part: 



— L?i ~ 



LjLo 



Lh such that L = (L?i,Lo)^ 



or vice versa 



Lo = Lo — 



IjT L 

— -Lo such that L=(L^,Lo)^ 

\LhF 



It is advisable to weight both parts equally by using the bisector of and Lh 
respectively Lq and Lq- In equation (3.4.3) this is achieved by the factor i. 



3.5 Estimating Projective Transformations 

So far we have looked at ways to estimate geometric entities such as points, 
lines and planes from observations that were related to the unknown in some 
way. We now want to focus on estimating transformations, a common tech- 
nique using point observations is the so-called DLT algorithm, see below. We 
will interpret a transformation of entities as a trilinear relation between the 
original entity, its image entity and the transformation itself: then it is easy 
to derive several kinds of DLT algorithms. 

3.5.1 Collinearity Equations 

The collinearity equations are the classical equations in Photogrammetry and 
are derived from a pinhole-camera model with a given inner orientation, let’s 
say only the focal length c, a rotation matrix R = (rij) and a projection 
center (Xo,Yq, Zo)~^: 

rii(X - Xq) + ri2(y - To) + r^sjZ - Zp) 

^3l(X — Xq) + r32(y — ho) + T3z{Z — Zq) 

r2i(X - Xq) + r22{Y - Yp) + r23{Z - Zp) 

^ fSliX — Xq) -I- r32{Y — ho) + f33{Z — Zq) 



(3.89) 
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We can include more parameters for the inner orientation to this basic equa- 
tion, specifically skew Cs, scale difference m between the axis and the principle 
point {xh, Vh)^ ■ Then we obtain the Euclidean interpretation of a projective 
camera matrix P as in equation (2.22), which can be written as 



, piiX + P12Y + pi^Z + pu 

X — 

PziX + P32Y + P33Z + P34 

, P21X + P22Y + P23Z + P 2 i 

y = 

P31X + P32Y + P33Z + P34 



The abstract formulation of the collinearity equations have been described 
by Das 1949 and Abdel-Aziz and Karara 1971. 

In case a set of corresponding or homologous pairs of image points x'^ and 
Xi are known and the camera parameters pij are unknown, we can obtain 
Pij by multiplying both equations in (3.90) with their common denominator 
and transforming it into a set of homogeneous equation: 



(-X, -Y, -Z, 1 0 0 

Vo 0 0 0 -A, -y. 



0 0 x'A, x[Y, x\Z, xA 

-Z, 1 y\X, y[Y, y[Z, y'J 



( pii\ 

P 12 



= 0 



\P 34 J 

(3.91) 

The equation is linear in the unknown parameter vector p. Since it is of a 
form Aip = 0, it can be solved if at least 6 homologous point pairs are given 
by minimizing 

(3^ ( ^ AJA, J (3 min 



with the constraint such as f3~^ f3 = 1. One can use eigenvector solution as 
above, equation (3.86), advisably implemented using a singular value decom- 
position 

This method for computing projection matrices is known as the direct linear 
transformation algorithm (DLT) since the equation (3.91) is linear in the 
unknown vector. It actually works not only for the projective camera matrix, 
but for any general projective transformation, including homographies and 
even affine transformations. A reference for the DLT in the context of com- 
puter graphics and vision is Sutherland 1963, though Das 1949 already 
describes the technique. It was named DLT by Abdel- Aziz and Karara 
1971. 



3. 5. 1.1 Collinearity Equations Interpreted as Geometric Relations 

One can geometrically interpret equations (3.89) by the fact that the image 
point a;) on the camera plane, the projective center and the object point Xi 
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should be collinear, hence the name of the equations. Using this interpreta- 
tion, the collinearity equation are nothing but an algebraic expression for a 
geometric relation involving the image point, the object point and the cam- 
era. With the algebraic expressions for constructions and relations introduced 
in this chapter, we can reformulate this collinearity relationship; using the 
homogeneous vectors x' and the reformulation can be done in the image 
plane (2D) as well as in object space (3D): 

(2D) The object point is collinear to the image point x' on the camera 
plane, if the projected point PXi is identical to x', thus if the 2D 
relation x^ = PX^ holds. Algebraically, we obtain the bilinear forms 
(table 3.5): 



S(x')PX, = 0 (3.92) 

-S(PXi)x' = 0 (3.93) 

The above expressions are linear in X^ and x(. Using proposition B.5 
on page 185, we can reformulate equation (3.92) such that the same 
condition is linear in the parameter vector p = vec(P^) of the projective 
camera: 

S(x')(/ 3 ®Xj)p = 0 (3.94) 

Note that the tests for identity of two 2D points has r = 2 degrees of 
freedom. Thus one can use the reduced matrix S[^l(xi) instead of S(xi) 
in the last equation (3.94). If the first two rows are kept in S[^l(xi), the 
resulting equation is equivalent to (3.91). Thus we have easily derived 
the formula for a DLT solution by using the algebraic expression for 
x' = PXi, cf. Hartley and Zisserman 2000, p. 71f. 

We obtained a trilinear expression for the collinearity equation, thus 
we sometimes write the collinearity equation as a relation with three 
entities, = (x';p,Xi), where the vector p = vec(P^) represents the 
projective camera. 

(3D) Another option to express the collinearity relation is to check whether 
the viewing ray L'(x') of x' goes through the entity point Xi; in other 
words: the collinearity equation holds if the 3D relation X' G L'(xi) is 
true, which is equal to 

r^(Q^x')X, = 0 r^(QTx')X, = 0 (3.95) 

TT^(X,)Q^x' = 0 TT'^(X,;)Q'rx' = 0 (3.96) 

with L'(x') = Q^x' = CQ^x'. cf. equation (3.48). Again, we can make 
the expression linear in the camera parameters, this time for the line 
projective camera q = vec(Q^): 



Tr^(X,)(x'/ 0 /g)q = 0 



(3.97) 
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Observe that this time, the expression is linear in the parameters of the 
line projection matrix Q. The check for incidence of a line and a point 
in 3D has r = 2 degrees of freedom, thus in 3.97 we can use the reduced 
matrix to obtain two linearly independent equations, as ex- 

pected. Again we denote the relationship with the triple G (Xi;p,x'), 
where the vector p represents the projective camera and q = Jqp(p)p, 
cf. equation (3.51) on page 59. 



3.5.2 Coplanarity Equations 

One can not only estimate a projective camera matrix by homologous point 
pairs (x',Xi), but also by homologous line pairs (l',Li). For this case we can 
formulate a coplanarity relation^ for homologous line pairs: the projection 
center, the image line 1( on the camera plane and the 3D line must be 
coplanar. As in the last section, we can express this relationship in two ways: 

(2D) For homologous line pairs, the following identity relation in 2D must 
hold: 1( = QL, which is equivalent to the trilinear form: 



S(1')QL,= 0 


(3.98) 


-S(QL,) 1' = 0 


(3.99) 


S(l')(/3 0L7)q = 0 


(3.100) 


cf. table 3.5 and proposition B.5. We can also use 


the reduced matrix 


SM(l') with r = 2 to obtain a set of linearly independent equations for 


the relation = (1(; p, L) 




In 3D, the coplanarity condition for homologous line pairs uses the 


viewing plane of the image line: G A'(l'), which 

trilinear form: 


is equivalent to the 


rT(L,)pT i( = 0 


(3.101) 


-rrT(pTi,) L,= 0 


(3.102) 


rT(L,)(i'J 0 4) P = 0 


(3.103) 



The incidence of a line and a plane has r = 2 degrees of freedom, thus 
we can use F^^^^(Li) in the last equation. This establishes a trilinear 
equation for G (Li;p,x'). 

As for the collinearity relations, we obtain two trilinear forms, one expressed 
in 2D and one in 3D. But this time, the vector p from the point projection ma- 
trix P appears linear in the 3D case, while the vector q of the line projection 

® Note that the coplanarity relation is not related to the coplanarity condition 
that was formulated in the context of two images: the condition of coplanarity 
included the projection centers of the cameras and an object point. 
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matrix appears linear in the 2D case. Thus a DLT algorithm which wants to 
estimate the projective camera matrix P should use equation (3.103), which 
is the analogy to the equation (3.94) 



Table 3.8. Trilinear relations between the projective camera P and points and lines. 
Note that there are two possible expressions for point- resp. line-projection: the hrst 
expression constrains the entities in the image plane, thus testing if the projected 
point resp. line is identical to the observed one = (x';P,X) resp. = (l';Q,L). 
The other expression refers to the object space and tests whether the 3D entity is 
incident to the back-projected line resp. plane, € (X;P,x') resp. = (L;Q,1'). Note 
that q = vec(Q^) = JqpP with p = vec(P^), cf. equation (3.51) on page 59. 



transformation 


relation 


trilinear expressions 


projection 
x' = PX 


= (x';p,X) 


S(x')P X = 0 


-S(PX) x' = 0 


S(x')(/3®X'r) p = 0 


e (X;p,x') 


r'^(Q’^x') X = 0 


TT'T(X)Q'r x' = 0 


TT'r(X)(x'’^(g)/6) q = 0 
■lT'r(X)(x''^(g)/6)Jqp p = 0 


projection 

1' = QL 


= (i';p,L) 


S(1')Q L = 0 


-S(QL) 1' = 0 


S(1')(/3<8)L^) q = 0 
S(l')(/3®L'r)J,p p = 0 


e (L;p,n 


r'^(L)p'^ 1' = 0 


TT'^(P'^l) L = 0 


r(L)(l''^(g)/4) p = 0 



3.5.3 Simultaneous DLT with Points and Lines 



A DLT algorithm which simultaneously uses homologous points and homol- 
ogous lines is now quite easy to describe using equations (3.103) and (3.94): 
then we have equations of the form A^p = 0 and B^p = 0 which can be 
solved by minimizing 



12 = / 3 ^ 






(3.104) 
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with the constraint /3^/3 = 1. In general, any mixture of point and line obser- 
vations is allowed as long as there are 6 homologous pairs, as each pair adds 
2 constraints to the 11 unknown projective camera parameters (12 unknown 
entries of matrix P minus 1 homogeneous constraint). 

Note that one might also want to use the equations (3.100) and (3.97) using 
q to obtain an estimate of the projective camera, but as q is a 18-vector 
which represents 11 parameters, this is computationally not advisable. One 
may also use the Jacobian Jqp = 9q/5p to transform the equations (3.100) 
and (3.97) back to p, but then the resulting equation is not linear in p, cf. 
table 3.8. 

Our approach of simultaneously using points and line correspondences is 
similar to the work of Hartley and Zisserman 2000 and Luxen and 
Forstner 2001: both represent an observed 3D line Li indirectly by two 
incident points , then the coplanarity equation can be written as 

A'(1')Tx, . = I'JPX, . = 0 with i = 1, 2 

In our approach though, a 3D line is directly incorporated in the DLT by 
their homogeneous 6-vector. This has the conceptual advantage that both 
points and lines are treated in the same way: the formulas can be directly 
used if the 3D lines are given by their homogeneous 6-vector or if they are 
constructed from other entities — one does not have to choose incident points 
representing the line. 



3.5.4 Simultaneous DLT Algorithms for Homographies 

We now develop a DLT algorithm for estimating homographies in 2D and 
3D: given a set of observed entities and the corresponding transformed 
entities x(, which are supposed to be images of an unknown homography, 
thus ideally x/ = Hx^, where the tilde denotes true but unknown values. 

3.5.4. 1 2D Homography For the 2D case, we assume a set of homologous 
point pairs (x^, x'), * = 1, • • • , ria, and a set of homologous line pairs (Ij, 1' ), 
j = 1, • • • ,71;, for which the triple relations = (x'; H, x^) and = (1'; H, Ij) are 
supposed to hold. Identity in 2D has 2 degrees of freedom, so each homologous 
pair adds 2 constraints on the unknown homography H. There are 8 unknown 
parameters for H (9 matrix entries less 1 homogeneous constraint), so we need 
at least 2ux + 2ni > 8 equations or n = + ni >4 homologous pairs to 

estimate the homography - in case of n = rix + ni = 4, the homography 
is uniquely determined. For the triple identity relations, we can easily write 
trilinear algebraic expressions using the S(-) construction matrix, analogous 
to the 2D case of the collinearity and coplanarity relations. The resulting six 
equations for point and line pairs are listed in table 3.9, which can be used 
in a minimizing approach as in equation (3.104). 
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Table 3.9. Trilinear relations between 2D and 3D homographies and homologous 
geometric entities. The last row of each relation yields the DLT for the homogra- 
phy H. 



transformation 


relation 


trilinear expressions 


2D homography 
x' = Hx 


= (x'; H,x) 


S(x')H X =0 


— S(Hx) x' =0 


S(x ')(/3 (g) x^) vec(H^) = 0 




S(1)H't 1' =0 


-S(H'^l') 1 = 0 


S(l)(l'^ (g) k) vec(H^) = 0 


3D homography 

X' = HX 


= (X';H,X) 


TT(X')H X =0 


TT(HX) X' =0 


TT(X')(/4 ® X'T) vec(H'T) = 0 


= (L';H,L) 


(f^(L')H (g) /4)Jr L =0 


(/4(g) r(L)H'^)JrC L' =0 


(f^(L')(g) r(L)) vec(H'T) = 0 


= (A';H,A) 


Tf(A)H'^ A' =0 


Tf(H'^A') A =0 


TT(A)(A''^ (g) k) vec(H'^) = 0 



3. 5. 4. 2 3D Homography To estimate a 3D homography, we can have three 
different homologous sets: homologous point pairs (Xi,X'), i = I,- - • ,nx, 
homologous plane pairs (Aj, A'), j = 1, • • • ,n^ and homologous line pairs 
(Lfc, LJ.), j = 1, • • • , riL, such that the triple identity relations = (X'; H, X^), 
= (A';H,Aj) and = (L5.;H,Lfc) hold. A 3D homography H contains 16 
matrix elements, thus 15 free parameters. Therefore we need 3nx + + 

Sua > 15 equations, for example at least 3 lines, 1 point and 1 plane in 
general position, which in this case uniquely defines the full homography. 
For the identity relations of points, the trilinear algebraic expressions are 
obtained using the bilinear form TT(X')HXi = — TT(HXi)X' = 0 and propo- 
sition B.5 on page 185. The algebraic expressions are listed in table 3.9. 

For planes, one uses A' = H "'"Aj H^A' = Aj to obtain the trilinear 
forms, again listed in table 3.9. 

The algebraic expression for the identity of homologous 3D lines (Lfc,L'j.) 
is a bit more involving, as the identity expression is based on a product of 
Pliicker matrices, D = r(L')r(L) = 0, cf. equation (3.67) on page 74. The 
goal is to find an expression for vec(D), that is linear in L*,, L). and vec(H^). 
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Unfortunately, this is impossible: the 3D line homography Hi, defined in 
equation (3.41) on page 57 is quadratic with respect to the parameters of 
H. Thus an algebraic expression for the identity of L). = HiL^ can not be 
written in a linear form vec(D) = U(L(,, L)vec(H^) = 0. 

The solution to this problem is to choose a different distance matrix D', which 
is based on the interpretation of the rows resp. columns of the Pliicker matrix 
r(L): As described in section 3.2, especially table 3.3, the columns can be in- 
terpreted as four canonical points nEg, s = 1, • • • 4 that are incident 

to the line L. We can transform these points by the point-homography H and 
obtain four points Y(, compactly written as Hr(L/j). Assuming L). = H^Lfc 
the transformed canonical points are incident to the L),. 

Note that although Hr(Lfc) is a 4 x 4 matrix, in general this is not a valid 
Pliicker-matrix - but nevertheless the transformed canonical points can be 
tested to be incident to the line L), by r(L(.)Y( = 0. Thus we obtain an alge- 
braic expression for = (L'UH,L,), which is not quadratic in the parameters 
of H: 

= (L'fc;H,Lfc) D'=f(L'fc)Hr(Lfc) = 0 (3.105) 

D" = r(Lfe)HT(L'fc) = 0 (3.106) 

The second expression for D" in (3.106) uses the same reasoning as described 
above, but this time by transforming the canonical planes of the image lines in 
r(K) instead of the canonical points in T(L(,). Alternatively equation (3.106) 
can be simply obtained by transposing (3.105), thus D" = D'^. 

Using D" and the matrix J|- with vec(F(L)) = J|-L defined in equation (3.68) 
on page 75, it is now easy to derive the trilinear expressions for homologous 
line pairs, see table 3.3. 

Summarizing the previous results, we have equations of the form A^h = 0, 
Bjh = 0 resp. Gfch = 0 depending on homologous point, plane resp. line pairs 
that are linear in the parameters of the 3D homography vector h = vec(H). 
Thus minimizing 



12 = /3M ^ AjA, + ^ BjB, + ^ GjGfc 



/3 



(3.107) 



under the constraint /3^/3 = 1 yields an estimate for the unknown homogra- 
phy. 



3.5.5 Estimating Constrained Transformations 

Up to now we have constructed projective transformations without any con- 
straints on the parameters other than the homogeneous constraint. We want 
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to make a short comment on estimating special homographies, such as affin- 
ity or similarity but also the affine camera P“ without providing an explicit 
algorithm. 

(1 l) (2 3 ) 

In 2D, an affine transformation Ha is defined by the upper two rows HJi 
of a 3 X 3 matrix, the third row being (0, 0, 1)^, cf. equation (2.21) on page 35. 
This is a linear constraint on the unknown parameter vector. If we assume 
a similarity transformation, we even have two quadratic constraints, cf. sec- 
tion 2.3.1. 

Non-linear constraints can not be as easily incorporated as previously de- 
scribed when minimizing the algebraic distance. But one can use a two-step 
procedure similar to the estimation of Pliicker coordinates, cf. section 3.4.3: 
first compute a homography H as in the previous section. In a second step 
the additional constraint has to be enforced by choosing the “closest” affine 
transformation Ha to H. The closeness term is not uniquely defined and may 
underlie different criteria. In the case of affinity one could just skip estimating 
the first two elements of the third row; the resulting 7 parameters can again 
be written in a homogeneous vector. For similarity and other special transfor- 
mations, one need more sophisticated methods, which will not be discussed 
further in this work. 

In the field of Computer Vision, the two-step method is well-known, es- 
pecially within the task of estimating the fundamental matrix from point- 
correspondences, cf. Hartley 1995b. In this work the 8-point algorithm is 
discussed: first derive an equation of the form (x' ® x")f, with f = vec(F^), 
which can be regarded as a DLT for the fundamental matrix. The equation 
is solved by algebraic minimization and in a second step the singularity con- 
straint is enforced by minimizing the Frobenius norm, cf. Tsai and Huang 
1984. An overview of methods for estimating the fundamental matrix can be 
found in Zhang 1998 and ToRR AND Murray 1997. 

3.5.6 Conditioning of Entities for Minimization 

As mentioned in the last section. Hartley 1995b used the estimation of 
a fundamental matrix as an example for a DLT and investigated the effect 
of (homogeneous) transformations of the 2D image points to the precision 
of the solution. Indeed, given two related sets of homologous, but slightly 
inconsistent image points (x',x") and (Tix(, T 2 X"), then the fundamental 
matrix F for the first set is numerically not equal to Tj"^FT 2 , as different 
algebraic errors are minimized. 

It turns out that it is very much advisable to “normalize” the points such that 
their centroid is at the origin and the points are scaled so that the average 
distance from the origin is equal to -\/ 2 . Because of the improved condition of 
the design matrix, the stability of the linear equation drastically improves. 
As we assume that the coordinate are approximately close to the centroid, we 
concentrate here on the scaling operation: it has already been introduced in 
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section 2.4 and was called conditioning of geometric entities. Thus contrary 
to Hartley 1995b, we will call the coordinate transformation conditioning 
of the geometric scene to avoid confusion with the normalization operation 
of homogeneous vectors introduced in section 2.1.1 on page 21. 

The same positive effect of conditioning occurs not only when estimating 
the fundamental matrix, but also when we construct unknown entities based 
on known related entities: as an example, assume four observed 2D lines b, 
where each of them is either parallel or orthogonal and should be incident 
to an unknown point x = (t, t, 1)^, cf. figure 3.11. Each line has an error- 
distance e from the unknown point. If e = 0, we get an exact solution by 
minimizing (3.86) with Ui = ij. Numerically though, the matrix U = IjJ 
becomes ill-conditioned when the point x moves away from the origin, using 
the ratio of the two largest singular values as the condition number (note 
that ideally, the matrix U has rank 2). This is getting worse when e > 0. By 
translating the centroid of the lines, x' = (0, 0, 1)^ and scaling the entities 
such that e' < 1, the matrix U' remains well-conditioned. 
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Fig. 3.11. Left: this is an example of an estimation error when not conditioning a 
geometric scene: the goal is to construct an unknown point from four supposedly 
incident lines. Right: The estimation error is plotted in percentage depending on 
the distance of the unknown point from the origin. The error is small if the point is 
close to the origin or far away. The error is plotted for various conditioning factor 
/. The error is small for / = 0.1 independent of the distance of the origin. 



In section 2.4 an algorithm for conditioning homogeneous vectors of geometric 
entities was proposed, which is similar to the one mentioned in Hartley 
AND ZiSSERMAN 2000, p. 113. For the purpose of estimation of geometric 
entities, a conditioning factor fmin of the order of 0.1 was empirically found 
to be useful when applying the algorithm 3.3, which is introduced in the next 
section. 
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Algebraic optimization 

Objective: find an estimation for an unknown geometric entity /3 
using a set of observed entities yi, that are supposed to be related 
to the unknown entity by the geometric relations Ri{yi,(3). 

If the unknown /3 is a transformation, then the observed entities 
consist of corresponding pairs of entities {yi,y'i), such that = 
M(/3)yi with M(/3) being the matrix representation of the unknown 
transformation. 

1. If necessary, condition the observations yi with a conditioning 
factor / determined as in algorithm 2.1 with fmin = 0.1. For 
transformations, obtain two different factors /, /^ see equa- 
tion (2.25) on page 39. 

2. For each yi, obtain the construction matrix Ui, depending on 
the relation Ri and the type of the unknown entity /3, cf. ta- 
ble 3.5. For unknown transformations, check the tables 3.8 and 
3.9. 

3. For U = EiUlUi, compute a singular value decomposition 
U = EDG^. The estimate (3 is equal to the last row of G. 

4. If necessary, recondition the estimated (3 by 1//; for a trans- 
formation (3 use equation (2.25) with f,f. 



Algorithm 3.3: Algorithm for directly computing unknown geometric entities and 
transformations defined by a set of observations, that are related to the unknown. 



3.5.7 Generic Construction Algorithm 

To summarize the section on construction and estimation we introduce a 
generic construction algorithm, which can be used for estimating geometric 
entities in 2D and 3D, but also for projective transformations. Note that 
this algorithm can even be taken for computing unique constructions, thus 
estimations with zero redundancy, which is a bit more involving than the 
unique construction algorithm. 

In the next chapter we describe a statistical version of this algorithm, for 
which examples will be shown on artificial and real data. 
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In the last chapter we have developed simple algorithms for geometric rea- 
soning on points, lines and planes in 2D and 3D. They were designed with 
a perfect world in mind: we have not explicitly represented possible errors 
in the coordinates of the geometric entities. Only for optimal estimations of 
objects we have allowed the entities to have small errors, but have not given 
any error measures for observations and unknowns. The weakness of having 
no error representation becomes apparent when testing relations between en- 
tities, see section 3.3: to check a geometric relation, a distance measure has 
to be equal to zero, but in practice, the computed distances are only “almost 
equal to zero” . 

In part this is due to the finite precision of machine floating operations, but 
the real reason is that we usually deal with imperfect data: geometric entities 
are initially obtained by some measurement proeess. It is obvious that, the 
result of a measurement is generally corrupted by the imperfection of the 
measuring device, influences from the environment or the experience of the 
human operator. Additionally, the data model that is used is always imper- 
fect and does not reflect the reality appropriately. All these influences add a 
certain amount of noise or uncertainty to the true, ideal values and thus we 
only observe a distorted view of the ideal data. For practical applications it is 
of most importance to take the uncertainty of the observed data into account. 
The main question that is investigated in this chapter is how to represent the 
uncertainty such that the noisy nature of the data is considered appropriately 
and at the same time ensure that the complexity of the reasoning algorithms 
from the last chapter remains tractable. 

The outline of this chapter is as follows: we first develop an uncertainty repre- 
sentation of geometric entities and analyze the necessary steps for geometric 
reasoning with respect to the uncertainty. As the entities are parametrized 
by homogeneous vectors, the uncertainty representation includes entities at 
infinity. To keep the transformations during the reasoning steps sufficiently 
simple, it is practical to use an approximate representation of the uncertainty 
based on the second moments. We explore the errors of the approximate rep- 
resentation and the conditions under which the approximation is valid. The 
last three sections of this chapter use the results of the error analysis and 
introduce relatively simple and generic algorithms for constructing new un- 
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certain entities and test relations between them. We have implemented these 
algorithms in a software library called SUGR (Statistically Uncertain Geomet- 
ric Reasoning) and demonstrate the validity with various Monte Carlo tests. 
Note that we do not explicitly cover the uncertainty of line segments and 
planar patches. It is possible to represent segments and patches by their 
geometric entity - either an infinite line or a plane - and e.g. their bordering 
elements, endpoints and border lines, see chapter 2. 



4.1 Representation of Uncertain Geometric Entities 

In the last chapter we have derived ways to do geometric reasoning with ho- 
mogeneous vectors that represent points, lines and planes. This was achieved 
by doing both the constructions and the tests for relations completely in 
the homogeneous vector space. Only when interpreting the homogeneous co- 
ordinates in Euclidean space a vector was normalized by the norm of its 
homogeneous part, cf. section 2.2 on page 24. The transfer of the Euclidean 
parameters to the homogeneous vectors can be done by a simple transforma- 
tion. 

In this context, we now introduce the notion of uncertainty and develop 
an approximate representation for uncertain geometric entities within the 
framework of projective geometry. 

4.1.1 General Representation 

The desired uncertainty representation has to accomplish the following tasks: 

- transfer the Euclidean uncertainty to the homogeneous uncertainty as all 
imprecise measurements originate in Euclidean space 

- derive a new homogeneous uncertainty after constructions 

- normalize the homogeneous uncertainty back to the Euclidean space, since 
only the Euclidean interpretation of uncertainties are meaningful for Eu- 
clidean reconstruction. 

To model the uncertainty of homogeneous vectors, we assume each homo- 
geneous coordinate to be a continuous random variable, x = (^, • • • , XnY , 
such that we can define a probability density function (pdf) for homogeneous 
vectors: 

Definition 7 (Probability density function) The probability density 
function of a homogeneous vector x S is a non-negative valued 

function p : ^ R+ such that 
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The probability P(x G S) that a point x is within a region S C is 

P(x gS)= [ Px(y)rfy 
JyeS 

Note that the pdf of x is written as Px, where the index identifies the random 
variable, cf. Papoulis and Pillai 2002; sometimes we write x ~Px(x). 

A statistical projective point of dimension n may now be defined using its 
homogeneous vector representation x G with its distribution Px(x). 

Examples of pdfs for projective points are depicted in figure 4.1 for P^: the 
pdf Px(x) has an arbitrary form, whereas the pdfpy(x) is only non-zero along 
the Euclidean line parallel to w-axis. One can argue that both pdfs represent 
the same uncertain projective points: only the directional variation of their 
homogeneous vectors refer to Euclidean uncertainty, thus we may project the 
pdfs onto the unit sphere, then = py^(x). 




Fig. 4.1. Examples of uncertain 
homogeneous vectors in with 
different pdfs. The two pdfs Px 
and Py represent the same uncer- 
tain Euclidean entity, since their 
normalized pdfs are Pxa and py^ 
are identical. If the pdf is only 
non-zero along the direction of 
its vector as for pz, the underly- 
ing geometric entity has no errors 
and Pz is not a proper pdf, see def- 
inition 8. 



Thus analogous to the the equivalence relation (2.2) on page 22 , we can 
resolve the ambiguity caused by the homogeneity by defining 

X = y px, = Py, with x^ := N^(x) = and (4.1) 

Thus two stochastical homogeneous vectors are equivalent if their correspond- 
ing pdfs of the normalized vectors are identical, where the new pdfs p^, and 
Py„ are obtained by error propagation. 

Considering a homogeneous vector is x = (A®^: A)^j the homogeneous factor 
A is a non-stochastic value, but we may still regard it as a random variable 
A with a Dirae delta funetion as pdf, p\ = J(A — x). Thus we allow gen- 
eralized functions as pdfs. This has the effect that the equivalence relation 
of (4.1) might yield generalized functions as well. Therefore we reformulate 
equation (4.1) by using the cumulative pdf (cpdf) Px(y) = /x>yPx(x)dx 
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X = y <t4> Px 3 = Py, with N^(x) = and Ns(y) 

“ (4.2) 

Proper Density Functions As we have pointed out earlier, we assume that all 
free parameters of our underlying geometric entities are uncertain, thus the 
marginal distributions of the free parameters are not generalized functions. 
For an uncertain homogeneous vector x, this means that orthogonal to the 
direction of an observed x all marginal distributions must not be a Delta 
function. Technically, this means that the following property should hold: 

Definition 8 (Proper pdf for geometric entities) 

Given an observation x = {x\, ■ ■ • ,x„) from a pdf p^, an orthogonal trans- 
formation yf = Mx exists such that x maps onto (1,0,--- ,0)^. If for all 
i = 2, • • • ,n the marginal distribution for x\ is not a generalized function, 
then we call the Px a proper pdf for x. 

Looking at figure 4.1, definition 8 states that the pdf for v' must not be a 
Delta function, Pv'{x) yf 5{x). 

The above property is not sufficient for all geometric entities, though: as a 
3D line is represented by a 6-vector, but has only 4 free parameters, we have 
to take the Pliicker condition into account; the pdf for the dual line L also 
has to fulfill property 4.1 as it is a homogeneous vector. Thus the length of 
the vector is L = CL is not relevant to the free parameters of a 3D line. 

Definition 9 (Proper pdf for 3D lines) Given an observed 3D line L = 
(Li,--- , Ln) from a pdf pi^, an orthogonal transformation L' = ML exists 
such that L maps onto (1, 0, • • • , 0)^ and L maps onto (0, 0, 0, 1, 0, 0)^. If for 
all i = 2, 3, 5, 6 the marginal distribution for L[ is not a generalized function, 
then we call the pi^ a proper pdf for L. 

We conclude that a proper uncertainty representation of an observed geomet- 
ric entity is a pair (x, px), consisting of the observation x as an instance of 
the stochastical variable x and a probability density function px for x. The 
probability density function has to satisfy definition 8 resp. 9 for 3D lines. 

Transfer, Gonstruction, Normalizing We now explore the previously de- 
scribed steps for doing geometric reasoning, namely transfer of Euclidean 
entities, construction of new entities and normalizing back from homoge- 
neous to Euclidean entities, see also figure 4.2. 

- transfer 

We assume that a pdf Px is given for the Euclidean parameters contained 
in the uncertain vector x. The corresponding homogeneous vector is x = 
where A is the homogeneous factor which also has a pdf p\. 
Thus it is necessary to compute P(x,A)- 
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Fig. 4.2. General Statistical Geometric Reasoning using Homogeneous coordinates. 
As opposed to the diagram 3.1 in chapter 2, the observed entities are now repre- 
sented by its vector x and its pdf px. The task of reasoning now includes not only 
the transformation of the vector x including the propagation its pdf. With general 
pdfs, this is quite hard to accomplish. 



- construction 

As we have seen in the previous chapter, constructions of new entities given 
two homogeneous vectors are bilinear (or trilinear) functions z = z(x, y) . 
Thus it is necessary to compute the pdf Pz ■ 

— normalizing 

Normalizing back to Euclidean coordinates requires to compute the pdf 
p^o with yP = No(x) = |^- 

An additional remark is related to the term statistically well-conditioned, 
coined by Durrant-Whyte 1989 and which is unrelated to the term “con- 
ditioning” introduced in section 3.5.6. In few words, a parameter represen- 
tation is statistically well-conditioned, if the entity-representation relation is 
one-to-one and that the parameter vector of the representation must change 
“smoothly” with physical motion. Obviously homogeneous vectors do not 
satisfy the one-to-one condition. The proposed normalization of pdfs though 
ensures that for Euclidean or spherical entities the one-to-one relation can 
be established except for the sign ambiguity which is not important for our 
application. Therefore the comparability of two entities is still unique and 
applicable. 
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4.1.2 Approximate Representation 

It is fairly obvious that the reasoning scheme sketched in the last section is 
very hard to implement for general probability functions. Thus we need a 
special kind of probability function that is theoretically and computationally 
feasible. Because homogeneous vectors are normalized, it seems to be advan- 
tageous to use spherical distributions. There are many ways to statistically 
describe spherical data, cf. Fisher et al. 1987. One alternative used in Com- 
puter Vision is Bingham’s distribution, cf. Collins 1993. It is an antipodal 
distribution for directions on a sphere and is - similar to the Gaussian for 
the plane - a second-order approximation of symmetrical distributions on the 
sphere. This has the theoretical advantage of a real spherical distribution. On 
the other hand, operations on Bingham distributed variables in general do 
not yield a Bingham distributed result, additionally the formulas for doing 
reasoning may still be quite involving, cf. Collins 1993 for an example. 

We propose that for most applications it is not necessary to use a spherical 
distribution as long as the uncertainty is small, which will be demonstrated 
in section 4.3. 

4. 1.2.1 First and Second Moments and Gaussian Distribution As an 

approximation of general pdfs, only some characterizing parts will be used, 
specifically the first and second (central) moments of p^' 



The first moment is the mean value of x; if we have only one observed 
instance x, then it is reasonable to assume = x. The second central 
moment is the covariance of the coordinates Xi and Xj of x. The matrix 
■S'xx = (cij) is the covariance matrix of the uncertain homogeneous vector x. 
For the variances on the diagonal of Sxx, we use the notation af = an. 
Usually we have only one observation x available of a geometric entity. Then 
we postualte x to be the best estimators for the mean The proposed 
uncertainty representation for a geometric entity is now a pair (x, S^x) con- 
sisting of the actual observed entity, represented as a homogeneous vector x 
and the covariance matrix of the homogeneous vector. 

As we disregard all moments of higher order, we only approximate the ideal 
pdf. But if we assume a Gaussian distribution function 
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then the pdf is uniquely determined by the mean /x and the covariance 

S . If X is Gaussian or normal distributed, we write x ~ S). 

4. 1.2. 2 Singular Covariance Matrices In the following sections we will 
be confronted with cases where an uncertain vector x is assumed to be nor- 
mally distributed x ~ but S is singular, hence |X'| = 0; the co- 

variance matrix S is positive semi-definite instead of positive definite for 
1.571 yf 0. A singular covariance matrix indicates that at least one marginal 
distribution is not a proper function. As an example, assume y = {y^,y.^y 
with Syy = ( 0 o), then y^ ~ N{yy^,l) and y^ ~ N{y.y^,0), cf. figure 4.3. 

Fig. 4.3. Examples of uncer- 
tain homogeneous vectors in 
with different Gaussian pdfs, see 
also figure 4.1. The two covari- 
ances 57xx and Syy represent the 
same Euclidean uncertainty, since 
their normalized covariance ma- 
trices are identical. If the covari- 
ance matrix has a nullspace which 
is orthogonal to an observed vec- 
tor X, then it does not represent 
a proper covariance matrix. 

Although the covariance matrix X'xx may be singular, it has to fulfill the 
property that it refers to a full-rank covariance matrix of the parameters of 
the underlying geometric entity. We can reformulate definition 8 for general 
pdfs to a property of the covariance matrix: 

Proposition 2 (Proper covariance matrix for entities) Given an ob- 
servation X = - ,Xn) being an homogeneous vector and a covariance 

matrix .57xx- If the nullspace Af(X'xx) of X'xx does not intersect with the sub- 
space {x}"^ orthogonal to the vector x; 

Af(i:xx) n {x}^ = {0} (4.4) 

then the Gaussian with = x and is a proper pdf for x. We call X'xx 
a proper covariance matrix for x. 

Proof: Referring to definition 8, we look at the orthogonal transformation 
x' = Mx, such that x is mapped to e^ = (1, 0, • • • , 0)^. Using first-order error 
propagation, see page 104, the transformed covariance matrix is .57x'x' = 
M J7xxM^. For all coordinates x' the variances a^, must not be zero, only a^, 
for the first coordinate may be zero. In other words, the nullspace A/’(X'x'x') 
of Sx'x' may be empty or include e^. Transforming back using yields 
equation (4.4). □ 




104 4 Statistical Geometric Reasoning 



Thus a covariance matrix for homogeneous vectors has a rank of r > n, 
with X G Again, for the special case of 3D lines, definition 9 has to 

be reformulated as only 4 free parameters are contained in the six Pliicker 
coordinates: 

Proposition 3 (Proper covariance matrix for 3D lines) Given an ob- 
served 3D line L = (Li,--- ^ Lq) and a covariance matrix ^ll- If the 
nullspaceJ\f{SiAf) of does not intersect with the subspace {L,L}-^ or- 
thogonal to the vectors L and L; 

Af(i:LL)n{L,L}^ = {o} 

then the Gaussian with Pl = L ond is a proper pdf for L. We call X'll 
a proper covariance matrix for L. 

We are now prepared to define the uncertainty representation of a geometric 
entity based on homogeneous vectors: 

Definition 10 (Uncertain geometric entity) A point, line or a plane in 
2D or 3D has been observed and represented as a homogeneous vector x, cf. 
section 2 on page 19. We represent the geometric entity with its uncertainty 
or uncertain homogeneous vector as a pair (x, S^x) where the covariance 
matrix S^x is a proper covariance matrix according to propositions 2 resp. 3 
for 3D lines. 

Note that this definition handles entities at infinity in the same way as enti- 
ties, that have a finite distance to the origin. 



4.2 Transformation of Uncertain Homogeneons Vectors 

It is important to explore the effects of transformations to uncertain homoge- 
neous vectors. We will see that for our applications there are mostly linear or 
bilinear transformations, only normalizations are completely nonlinear. This 
section will describe the Euclidean representations of geometric entities and 
how to transform them to homogeneous representations and vice versa using 
first order error propagation. 

We will concentrate on geometric entities such as points, lines and planes. 
We only briefly comment on geometric transformations that are represented 
by homogeneous vectors. 

4.2.1 First Order Error Propagation 

One advantage of using first and second moments for characterizing the pdf 
of a stochastic variable is the simplicity for first order error propagation, 
cf. Koch 1999, p. 100: 
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Proposition 4 (Error propagation) Let x G be an uneertain ho- 
mogeneous vector with the covariance matrix o,nd let A he a fixed 

(m + 1) X (n + 1) matrix and b a fixed (m + 1) vector , then the covari- 
ance matrix Syy of the linear transformation y = Ax + b is given as 

Syy = M:^^A^ (4.5) 



Note that we have not assumed any pdf for x, thus the error propagation is 
true even if x is not Gaussian distributed. But an important observation can 
be made if y is known to be Gaussian: 

Proposition 5 (Error propagation of a Gaussian) If x is normally 
distributed, x ~ iV(/i^, X'xx), the linear transformed y is again normally 
distributed: 

y ^ iV( + b, AX'xxA^ ) 

As we have seen, some of our mappings are bilinear thus are not strict linear 
transformations as products of different coordinates appear in the formulas, 
cf. section 3.1 on page 48. In general we assume a real differentiable vector 
function f(x) of a homogeneous vector x S To approximate a function 

value y = f(x + Z\n) we can apply the Taylor series 

f(x + Z\n) = f(x) + f 1 Z\n + C>(Z\n^) (4.6) 



If the terms of second degree and higher, O(An^), are sufficiently small 
enough, we can approximate the covariance matrix Syy of y by 



S 



yy 



JiJyyjT 



with 




(4.7) 



With the first order error propagation, it is now possible to investigate in 
detail the three steps for doing statistical geometric reasoning with the pro- 
posed approximate representation: transfer of uncertain Euclidean entities to 
Homogeneous coordinates, uncertainty propagation for construction of new 
uncertain homogeneous entities and normalization back to uncertain Eu- 
clidean entities. In section 4.3 on page 113 we will discuss the validity of the 
approximation in equation (4.7) with respect to these operations. 



4.2.2 Transfer to Homogeneous Coordinates 

In this subsection we explore how to transform the uncertainty of a Euclidean 
representation in the proposed homogeneous representation. 
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4. 2. 2.1 Points Without loss of generality, we only deal with 2D points, as 
it is basically the same as the 3D case. Given a Euclidean 2D point x = 
{xi,X 2 )~^ € we have seen that a simple mapping yields a corresponding 
homogeneous vector x = {xi,X 2 , 1)^, see equation (2.3) on page 22. We can 
reformulate this mapping to 
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If Sxx is the covariance matrix for x, then the covariance matrix X'xx for x 



is according to theorem 4: 
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In general, a vector x^ = (Axi, Xx 2 , A)^ yields 
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The transfer of uncertain Euclidean points to homogeneous vectors is rigorous 
as the transformation is linear, no loss of information is involved. This is 
important as in most applications a reasoning process starts with points as 
observations, either in 2D or in 3D. 

We assume that only non-infinite or Euclidean points can be directly ob- 
served. Infinite points, such as points on the horizon in an image, do exist, 
but are usually constructed using Euclidean entities, such as using a projec- 
tive camera and a 2D image point. Thus the covariance matrix of an infinite 
point is determined by the covariance matrices of its constructing entities 
using error propagation. 



4. 2. 2. 2 Hyperplanes Hyperplanes are 2D lines and 3D planes. Again we 
concentrate on the 2D case: directly observed 3D planes are not common in 
the computer vision context, but 2D lines are frequently observed in images. 

Given a normal vector n = with Sn = ^ and a distance 

d with ffd and covariances aud and a^d the covariance matrix for l(n, d) is 
obviously 
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(a) point in 2D 



(b) point in 3D 



Fig. 4.4. Visualization of confidence region of an uncertain 3D point 



As observed in section 2.2.1 on page 25, there are lots of Euclidean rep- 
resentations for a 2D line and not all transformations to the homogeneous 
representation can be covered in this work. We particularly use one represen- 
tation of an uncertain 2D line, which is based on the point-angle form and 
consists of a 5-tuple 

as proposed by Forstner 1992 and used by Fuchs 1998. A detailed deriva- 
tion of this representation is found in the appendix C.1.1 on page 187. The 
point Xm = {xrm UmY the center point or point of gravity of the line, where 
the displacement error orthogonal to the line is minimal and equal to ad- 
The further away we move from Xm along the line by u, the larger is the 
displacement error, namely a-u = a'^u^ + a^. The resulting hyperbola in 

figure 4.5(a) can be interpreted as the union of all ID-confidence regions of 
points on the line, that are uncertain only orthogonal to the given line, see ap- 
pendix C.1.1. Note that a^p and ad are uncorrelated in representation (4.11), 
see section C.1.1 for details. 

The transformation from (4.11) to the proposed representation is 
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(4.12) 



with d = Xm cos{(fi) -I- ym sin((^). 

For a 3D plane, the confidence volume is a hyperboloid as shown in fig- 
ure 4.5(b). Similarly to the 2D case, there is a center point Xm, where the 
displacement error orthogonal to the plane is minimal. If the plane was con- 
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(a) line in 2D (b) plane in 3D 

Fig. 4.5. Visualization of the confidence region of a 2D line and a 3D plane. In (a) 
the confidence region is a hyperbola, with the center point being the point on the 
line with a minimal displacement error cr^ orthogonal to the line. The hyperboloid 
in (b) is the confidence volume for a plane again with a center point at the 
minimal displacement error. 



structed by three uncertain points, the center point is the weighted point of 
gravity of the three points. 

4. 2. 2. 3 3D Lines For 3D lines there are even more ways to represent the 
Euclidean parameters than for hyperplanes, but none of them are used fre- 
quently enough to serve as a reference. This is due to the fact that 3D lines 
are hard to observe directly; in computer vision applications they are usu- 
ally observed indirectly by two 3D points or by two image lines. For both 
cases we have provided simple algebraic formulas which can be used for error 
propagation, see section 4.2.6 on page 111. 




Fig. 4.6. Visualization of the conhdence region of a 3D line. Its shape is a twisted 
cone, which is bent at two locations. 



A visualization of the confidence volume of a 3D line is shown in figure 4.6, 
see also Forstner 1992. The shape of the volume is a twisted cone, which is 
bent at two locations. The shape can be explained as follows: a 3D line may be 
given by two 3D points, where the first point has an error ellipsoid elongated 
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along the x-axis and the second point has an error ellipsoid elongated along 
the y- axis, thus the two ellipsoids are twisted against each other. 

4. 2. 2. 4 Geometric Transformations We represent the uncertainty of pro- 
jective point transformations H by a covariance matrix J7hh of the vector 
h = vec(H^). Here, we assume that the covariance .57hh is a proper, cf. propo- 
sition 2. In case the transformation and its uncertainty is given in Euclidean 
parameters, the covariance can be built using error propagation. The func- 
tional relationship between the given Euclidean representation and the homo- 
geneous matrix is application dependent: it is not easy to describe a general 
method for establishing such a functional relationship: as an example, there 
are seemingly countless ways to describe the inner and outer orientation of a 
camera. 

Assuming that the covariance of a point transformation H is given. To obtain 
the covariances for hyperplane transformations Ha = H~^, one can com- 
pute (Forstner 2002) 

0 H"i)i:hh(H~i 0 H-T) (4.13) 

The covariance matrix for a 3D line transformation is also computed by 
error propagation using the Jacobians mentioned in (3.41) on page 57; for 
the projective camera matrix Q for lines, see (3.51) on page 59. 



4.2.3 Normalization to Euclidean Coordinates 

Transferring a homogeneous vectors x back to Euclidean entities was done 
using the normalization operation No(x) = x/|a;/i|, where the norm was 
skipped for point entities. We now want to extend the normalization operation 
to uncertain geometric entities (x, J7xx)- Again we apply first order error 
propagation and as the normalization is a non-linear function, we need to 
derive the Jacobians J = ■ The normalization operation Nq((x, Sy^^)) 

is then given by 

No((x, i:xx)) = f A’ J(x)i:xxJ^(x)) = (x', Sy.y,) (4.14) 

Note that the ambiguity in the sign of a normalization No(-) does not affect 
the error propagation. The Jacobians for 2D points and lines and for 3D 
points, lines and planes are listed in table 4.1. The nullspace of the normalized 
covariance matrix is the nullspace of the Jacobian J^, see table 4.1. Note 
that the nullspace is equal to the embedded homogeneous part of the vector; 
for example if 1 is a 2D line, the nullspace of Sy\i is {Ih, 0)^. 

It is possible to use the acquired pair No((x, X'xx)) to transform to any 
other Euclidean representation. For example, the transform for a 2D line in 
representation (4.11), see on page 107 is described in the appendix C.l. 
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Table 4.1. Jacobians for Euclidean normalizations No(-) for points, lines and 
planes in 2D and 3D. 
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4.2.4 Normalization to Spherical Coordinates 



We may also want to normalize a homogeneous vector to unit length, called 
spherical normalization, cf. page 22. Given any homogeneous vector x S P”, 
the Jacobian to the spherical normalization Ns(x) = |^ is 



J^(x) 



9N,(x) 

dx 




(4.15) 



Thus 

, J,(x)i:,jJ(x)^ (4.16) 

This equation holds for all geometric entities and transformations. It is easy 
to see that the nullspace of Jg and thus of the transformed covariance matrix 
is equal to x. 



Ng((x,i:xx)) = 



4.2.5 Changing Nullspaces Using Orthogonal Projections 



It is easy to see that the Jacobian Jg for the spherical normalization is essen- 
tially an orthogonal projection 'P onto the space {x}-^ (Koch 1999) and the 
appendix B.2 on page 184: 



Js 




with 'Z'x = / — x(x^x) ^x^ 



(4.17) 



4.2 Transformation of Uncertain Homogeneous Vectors 111 



Thus we may also apply only the projection matrix 'P to the covariance 
matrix. Then we only change the nullspace of the covariance matrix but 
don’t change the metric information. This is valid as long as we don’t impose 
a nullspace that is a subspace of {x}-*-, see definition 10 on page 104; for 
spherical normalization, the nullspace is x. Thus we define a new operator 
Vy for uncertain homogeneous vectors, which only changes the nullspace of 
the covariance matrix J7xx to y, but does not change the underlying vector: 



Vy ((x, i:xx)) := (x, 

with 'Vy being an orthogonal projection onto {y}'^ 



(4.18) 



For an uncertain 3D line L, we may project .E'll onto {L,L}^: 

^(L,L) ((L, ^ll)) := (l, 

with ^'(L L) being an orthogonal projection onto {L,L}'*‘ 

(4.19) 



Using a general dot-product x^Wy with W being a positive semidefinite 
matrix, one can also express the Euclidean normalization: with the general 
orthogonal projection from proposition B.4 on page 184, one can show 
that for any geometric entity x, 

with t^x.M = I - x(x'''Mx)”^x'''M (4.20) 

dx \xh\ 

where M is the identity matrix with zero entries on the diagonal elements, 
which correspond to the Euclidean part of the vector. 



4.2.6 Construction 



In chapter 3, constructions were shown to be bilinear operation, so two enti- 
ties X and y form a new entity z = f(x, y), which can be expressed as 

z = U(x)y = V(y)x (4.21) 



Assuming the entities to be uncertain, the pairs (x, S^x) and (y, Syy) and 
possibly the covariances between x and y, namely U^y are necessary for 
doing error propagation: 



(z, -S'zz) 



|^U(x)y, [V(y),U(x)l 





v^(y) 

UT(x) 



(4.22) 



In case of independence between x and y one obtains 

(z, i:,,) = (U(x)y, U(x)i:yyUT(x) + V(y)i:xxVT(y)) 



(4.23) 
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As an example, the uncertain 2D line (1, ^n) joining two points (x, X'xx), 
(y> ^yy) is given by 

(1, i:n) = (x X y, S(x)i:yyST(x) + S(y)i:xxST(y)) (4.24) 

with the skew-symmetric matrix S(x). Note that the error propagation is also 
contained in equation (4.18) of Kanatani 1996. 

Generally the covariance matrix S\\ has rank 3 unless the two points are 
identical. This is an example that we cannot assume that homogeneous co- 
variance matrices have a rank which is equal to the degree of freedom of the 
geometric entity. 

In this work, we usually assume that the geometric entities are independent 
from each other, thus equation (4.23) will be used for most of the following 
text. 



simple / rigorous by embedding covariance matrix 




simple / approximate error propagation 



Fig. 4.7. Approximate Statistical Geometric Reasoning using Projective Geometry. 
Gompared to figure 4.2, the pdfs are replaced by the covariance matrices of the 
vectors. The error propagation of covariance matrices is much simpler than for 
general pdfs. 



Summing up section 4.2, we are now capable of doing statistical geometric 
reasoning with unique constructions, as shown in figure 4.7 using an approx- 
imate uncertainty representations. In the next section we explore the errors 
that occur when using the proposed approximate representation. 
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4.3 Errors in Approximated Uncertainty Representation 

At certain stages of the reasoning process pictured in figure 4.7, we have 
mentioned approximations to the rigorous analytical method: an approxi- 
mated representation and approximation in error propagation. This section 
describes the nature of these approximation in detail. 

As spatial reasoning with homogeneous coordinates is based on bilinear forms 
(cf. table 3.1) and as normalization of homogeneous coordinates is a division, 
the analysis of the degree of approximation using homogeneous entities is 
much simpler than using only Euclidean entities, as has already been observed 
by Criminisi 2001 and Clarke 1998. 

We especially investigate the bias of the first-order error propagation, i.e. the 
difference between the true value and the approximated or computed value. It 
is important to remark that the bias has to be considered with respect to the 
measurement error that is to be expected: a bias that is 1% or even 10% of the 
true standard deviation is negligible as it has practically no consequences to 
the application. An example: assume that within a system that reconstructs 
buildings from images, a measurement error was estimated to be <7 = 2 cm. 
Then practically one needs not to care about the fact, that the true value is 
(T = 2.2 cm. 

We will see that - assuming the original errors are less than 10% - the 
bias is usually small or can be controlled in a simple way to obtain valid 
approximations of the true uncertainty. 



4.3.1 Second Moments and Gaussian Assumption 

Using the first and second moments of a pdf means that we use the Lapla- 
cian approximation by cutting off the Taylor series after the linear term. Of 
course, this causes systematic errors (bias) in mean and variance. But if no 
other information than the first two moments are available, then in general 
the Gaussian distribution is the most likely distribution due to the maxi- 
mum entropy principle Cover 1991. Additionally, the central limit theorem 
states that the average of a sufficiently large number of independent stochas- 
tical variables with identical standard deviations tend to form a Gaussian 
distribution. 

Previous work on uncertain geometry in the Computer Vision (Collins 
1993, Haddon and Forsyth 2001) state that the Gaussian distribution is 
not useful for geometric reasoning, especially when the uncertainty is large 
and the operations on the entities are highly nonlinear. We shortly discuss 
these arguments: 

- Large uncertainty 

Consider the sphere as a unique representation space for homogeneous 
vectors, then it is true that the Gaussian does not approximate well for 
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large uncertainties: since only the tangential plane at the observed point 
on the sphere is considered, the bias becomes large when the uncertainty 
of the error becomes large. On the other hand, the tangential plane may 
very well approximate the actual distribution on the sphere as long as the 
error is small enough: assume an angle oi Lp < 20° then sirup « p> with a 
bias of less than 1%. In our application we assume small Euclidean errors 
which result in small directional errors for homogeneous vectors^. 

In particular, we assume Gaussian distributions for Euclidean point coordi- 
nates and for Euclidean directional vectors, namely the homogeneous part 
of hyperplanes and both parts of the 3D line vector. While Gaussians are 
widely accepted for point coordinates, there exists many different distribu- 
tions for directional vectors (for example von Mises distribution, Bingham 
distribution). Again, we argue that for small uncertainties the bias of a 
Gaussian distribution compared to a spherical distribution is small. In the 
following subsections, we will see that an error of less than 10% is small 
enough to assume Gaussian distributions. 

- Nonlinear operations 

The error of only using second moments can be arbitrarily high when as- 
suming highly nonlinear operation on random variables. Thus in general, 
no assessment can be made for the accuracy of the approximation. The op- 
erations that we use in this work are bilinear operations on homogeneous 
vectors. In the next two sections, we will investigate the bias caused by 
the particular bilinear function from chapter 3. Again, we will see that for 
small errors in the observed geometric entities the bias can be neglected. 



4.3.2 Bias in Scalar Multiplication 

The error propagation for constructions as in equation (4.22) and (4.23) on 
page 111 is not rigorous since generally products of stochastic variables occur, 
thus being non-linear. 

The most simple bilinear operation is the product z = xyof two stochastic 
scalars x and y. This is the only non-linear form appearing in any unique 
construction using join or intersection, thus in the following it is explored in 
detail, x and y are assumed to be normally distributed by ax resp. ay and 
having a covariance axy 

The stochastical variables x, y are modeled by adding a stochastical noise 
vector e^,, Cy to the mean thus e.g. x = y{x) + e^,, where e^, ~ N{Q,a'^) is 
normally distributed. For z we get 

1 = Ki) +^z = ilJ-x + ex){Hy + Cy) = PxPy + PyCx + fJ-xCy + Cx-Cy (4.25) 

^ This is a similar argument as mentioned in the footnote 1 on page 19: it is 
commonly accepted to approximate the world as planar as long as we deal with 
man-made structures within an area of a few kilometers. 
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Note that equation (4.25) is equivalent to the Taylor series at the mean values. 
In this case the Taylor series is finite and we showed all terms. 

Using (4.25), the mean value U(z) is given by 

T/(z) = E[flxfly + fJ"yGx + fj-x§_y + = /ix/Ty + (Txy (4-26) 

The variance U(z) = E{^) — U(z)^ can be computed by 
T/(z) = ^x^^y 4" ‘^f^xfJ'yO'xy + B^xy 

E{^) = + l^Wx + + alal + 2aly + 4^xf^yaxy 

The last formula can be derived by expanding and using the equality 
E{elel) = alal + aly and thus 

V U) = ^l = + 2/ix/iyO-xy + cr^cr^ + a^y (4.27) 



E{z) and V(^) are analytically strict any distribution. Adapting equa- 
tion (4.23) on page 111, thus using first order approximation for mean and 
variance yield: 

E^^Hz) = fixfiy (4.28) 

y (z) = + 2fj,xHyaxy (4.29) 



Comparing first order approximation with the strict solution, the relative 
bias te and ry for variance and mean is given by 



hE A(z)-U«(z) 
te= — = 

z CTz 

_K _ u(z)-yW(z) 

_9 _9 



^ xy _ 
(Tz 



Pxy^x ^y 



xy 






Pxy^ 



(4.30) 

(4.31) 



We chose to normalize the bias by the standard deviation cr^ resp. variance 
(Tj to obtain an objective measure relative to the expected error 
If the correlation is zero, there is no bias in the mean , thus te = 0. In the 
case of no correlation, the relative bias in variance reduces to (T^CTy/cr^. In 
case of a = ax = cTy and axy = 0 the relative bias reduces to 



rv 



a 



2 



pI + pI + 



(4.32) 



Thus only if < 9 a^ (i.e. x and y are both expected to be very close 

to zero with respect to a) the relative bias in variance is larger than 10%. It 
is important to note that the bias is always positive, thus the true standard 
deviation is always larger than the approximated one. 

In case the correlation is non-zero, we consider the case oi y,x = Py = f 
and Ox = 0.05 and Uy = 0.2, therefore assuming an unusually large standard 
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deviation of 20% for y. The relative bias of the mean is less than 4%, the 
relative bias of the variance is less than 1 %, cf. figures 4.8(a) and 4.8(b). 
The influence of the variation of y-x and yy are depicted in figures 4.8(c) 
and 4.8(d): only if yx = yy < a, the bias becomes larger than 10 % for the 
variance and larger than 20% for the mean. 




(a) relative bias r_B of mean with re- 
spect to correlation p^y 





(b) relative bias rv of variance with 
respect to correlation p^y 




(c) relative bias r_B of mean with re- (d) relative bias ry of variance with 
spect to correlation^ and y^ = Py respect to correlationp and y^ = Py 



Fig. 4.8. Relative bias rs and rv of bilinear operations with scalar multiplication 
as an example. For figures 4.8(a) and 4.8(b) the means of a; and y were fixed to 
1, Px = Py = ^ and the standard deviations to = o-y = 0.1. The correlation 
factor pxy varies between -1 and 1. In 4.8(c) and 4.8(d) the mean of a; and y 
varies simultaneously between y^ = Py = [—2 • • ■ 2] and the correlation varies again 
between -1 and 1. The bias is usually around 2% or less and becomes intractable 
only if px = Py ~ 0, see text for details. 



From this analysis we conclude that for the most simple bilinear form - 
namely scalar multiplication ~ the bias is significant if the standard deviation 
of both numbers is larger than their mean. This has been described for more 
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complex bilinear forms by Haddon and Forsyth 2001. Usually in vision 
problems the relative accuracy is less than 10 %: Thus we do not expect 
significant problems caused by bilinear scalar multiplication. 



4.3.3 Bias in Bilinear Constructions 



Next we want to explore the bias for bilinear operations on homogeneous 
vectors instead of scalars. Again, we choose a simple example for a theoretical 
analysis. 

Assume two stochastic homogeneous vectors x, 1 S on the projective axis, 
where x = (u,u)^ represents a projective point and 1 = (a, 6)^ represents 
a projective hyperplane in P^, see figure 2.1 on page 21. Furthermore two 
covariance matrices Sxx and Su are given, which fulfill definition 10. 

As a bilinear operation we choose the dot-product d = x^l = l^x and are 
especially interested in the mean value E(d) and the variance ad of V{d). 
Assuming Gaussian distribution, we can compute the true values by refor- 
mulating the dot-product into a quadratic form: 



d = 



^Wz = {u,v,a,by- 



/o 0 1 o\ 




(A 


0 0 0 1 
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10 0 0 




a 


\0 1 0 0/ 




\b) 



Mean and variance of d are now given by 

E{d) = trace ( + /^I = au + hv (4.33) 

U(d) = 2 trace ( 

= + “^cTabCruv + <yl<yl + u^crl + 2uvaab + v'^al 

-\- a^ 2abauv + (4.34) 



cf. Koch 1999. As we assume x and 1 to be uncorrelated, there is no bias 
between the first order approximation E^^^d) = and E{d). The vari- 

ance obtained by the first order error propagation and the bias by with 
respect to the ideal case is 

^ = u a^ 2 iivaab T t a^ -t- a a^ -t- 2cibauy -t- b a^ (4.35) 

bv = V{d)~ V‘'^\d) = alal + afa^ + 2aabauv (4.36) 



It is easy to see that the addends in the bias of the variance are a consequence 
of the discussion of the last section, cf. equation 4.31. 
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Projecting the Covariance Matrix Unfortunately there is an important differ- 
ence to the last section: no statement can be made about the upper bound of 
the bias within this general setting. We may even obtain a zero-variance 
if the nullspaces of the covariance matrices are Af{Sxx) = 1 and Af(.£'n) = x: 
then the first order error propagation yields SxxfJ-i = 

0, independent from the ideal variance V (d). The difference to the last section 
is due to the fact that the covariance matrices and .5711 are not unique, 
the values of the matrices can be arbitrarily distorted as long as the nor- 
malized covariances matrices are the identical, N^j(J7xx) = Ns(.57n), see also 
figure 4.3. To avoid the ambiguity, we may normalize the covariance matrix 
prior to computing the bilinear operation. Since we do not want to change 
the vector x, we apply an orthogonal projection to the covariance matrix 
-S'xx = ^ cf. equation (4.18) and with some calculation one obtains: 

^ ^ ^ ^ {v'^crl - 2uvOuv + ( v'^ -uv 

{u^ + V^Y ^2 

and in the same manner to Si\. Note that the numerator of the factor in (4.37) 
is a quadratic form: 




- 2uvOuv + 




With the projected covariance matrices we obtain a new bias 



- 2uvauv 

by — 



v?' aYjQY <y^a ~ ‘^o.baab + + 2uvab + v?a?') 

{u^ + v^Yi^^ + b^Y 



(4.38) 

Although the new bias looks more complicated, we gained some new insights: 
first, the bias solely depends on the directional uncertainty of the homoge- 
neous vector, thus it depends solely on the underlying Euclidean information 
and can not be arbitrarily large. Second the bias by is non-negative, as the 
factors of the numerator can be written as a quadratic form of the covariance 
matrices, which are positive semidefinite by definition. Thus we obtain only 
optimistic approximations of the true variance. This is an important obser- 
vation which holds for all constructions with projected covariance matrices, 
as we will see in practical tests, cf. section 4.4.2 on page 125 and section 4.5.4 
on page 135. 



Conditioning the Vectors Although the bias was found to be non-negative, it 
may still be quite large: consider that the homogeneous vectors for the point 
and the hyperplane are identical, x = 1. Then N{Sxx) = x and N{S\\) = 1 
and V'^^Yd) = x^i7nx -f l^J7xxl = 0 and therefore the bias is maximal. 
On the other hand, assume the covariances were obtained by embedding the 
Euclidean covariance into a homogeneous vector, thus 



4.3 Errors in Approximated Uncertainty Representation 119 




If we don’t apply any projection on the matrices, the bias is zero, independent 
from the values and crp This suggests to transform an uncertain homo- 
geneous vector (x, (■S’xx)) with a proper such that the variance for 

the homogeneous part approaches zero. 

A simple way to achieve this goal is to condition the two geometric entities 
X and y according to the algorithm 2.1 on page 40: given a point x = (u, 
with a projected covariance matrix Vx (^xx), the variance (Jy of the homo- 
geneous part V of the vector approaches zero when conditioned with a factor 
/ < 1, cf. figure 4.9, since the conditioned and projected covariance matrix 
Vx° is close to For a hyperplane 1 , the same reasoning applies 

and the conditioned and projected covariance matrix Vio (i^ioio) is close to 
Sii- Note that in practice, we first condition the two vectors and then project 




Fig. 4.9. Effect of conditioning 
to uncertain homogeneous vec- 
tors x, y. Note that the covari- 
ance matrix of the condi- 

tioned vector x° is close to the 
Euclidean case if Af{Sx°x°) = 
x°, see text for details. 



the covariance matrix with Vx to avoid two projecting steps. This is due to 
the fact that the conditioning with the matrix \Nx{f) changes the nullspace 
in an undesirable manner and this has to be corrected. 

The reduced bias is dependent on the conditioning factor /: 

„ /"‘(au -h bv)'^{v'^al - 2uvauv + - 2abaab + 

(a2 + /262)2(J2^2 + .^2)2 

Note that by is already reconditioned by 1/ p to compare it with the previous 
bias. As an example assume a = 6= u = u = l, we then obtain 

~ (-J2 _|_ 1)4 = dUWv (4.40) 

such that / < 1 yield a smaller bias by than b'y, for / = 0.1, the reduction 
is 99.96%, see figure 4.10(left). The relative bias r'y = by/V{d) is shown in 
figure 4.10(right) and is about 2% for / = 0.1 and about 0.5% for / = 0.05. 
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Fig. 4.10. Left: Sample curve for the reduction factor g{f) between bias b'^r and by: 
by — g{f)"by, cf. equation (4.40). One can see that a conditioning factor / < 1 - 
thus scaling the Euclidean part down - greatly reduces the bias. A condition factor 
/ > 1 scales the homogeneous part of a vector down and the bias gets smaller, 
too, but slower than for / < 1. Right: This curve shows the relative bias r'y in 
percentage for ctJ dependent on / with d — x^l and x = 1 . 



The theoretical analysis of the bias for uncertain homogeneous vectors coming 
from can be generalized for P^ and P^, but is not elaborated in this work. 
We rather rely on the validation using Monte Carlo algorithms, see 4.4.2. An 
additional demonstration of the effect of conditioning/projecting for 2D is 
shown in the context of hypothesis testing, see page 132 



4.3.4 Bias in Normalization 



To investigate the bias of using first order error propagation in normaliza- 
tion, we only consider Euclidean normalization as defined in equation (2.14). 
The case of spherical normalization has already been informally discussed 
in section 4.3.1: as long as the uncertainty is small, a normalization to the 
tangent plane of a sphere is a valid approximation and the bias is expected 
to be small. 

For the Euclidean normalization, we again consider the most simple case: as- 
sume a point x G P^ on the projective axis, thus in homogeneous coordinates 
x = (it, u)^. Assume u and v being stochastical variables, with the covariance 
matrix 




The normalization x = Nq(x) = u/v is nonlinear and as in section 4.3.2 we 
consider the Taylor series: 



x = Rx + e^ 




flu fly 



l^U§Ly 



RuS.1 

rI 



+ + e„|^) 



(4.41) 



Contrary to the bilinear product the Taylor series is infinite and we consider it 
only up to second order to estimate the bias between first order and second 
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order approximation. This approach reflects the dominant error caused by 
considering first order approximation. 

The mean value and variance for the first and second order approximation 
are computed similarly to the last section and after some calculations one 
obtains 
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(4.44) 
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Thus we obtain the approximate bias b 
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(4.46) 

(4.47) 

(4.48) 

(4.49) 



The formulas for the bias are more complicated than for the scalar multipli- 
cation, but because of the factor one can see that the bias of both mean 
and variance is 0, if the homogeneous factor v is known to be certain. Thus if 
we normalize an embedded covariance matrix as in (4.10) on page 106 back 
to the Euclidean space, we obtain the same covariance matrix. 

Now assume that the vector x is conditioned, such that \hu\ < fmin\Hv\ with 
fmin < 1 and the covariance matrix is equal to its orthogonal projection: 
-S'xx = ’T^x(-^xx), cf. equation (4.37). Then cr„ is comparably very small 
compared to cr„, see also figure 4.9; therefore is close to a form as in (4.10) 
on page 106 and the bias must be very small. 

As an example, assume Hu = 10, /i„ = 1,(7^ = 0.5, cr« = 0.1. A plot of the 
relative bias rby'^ = by '^ depending on the conditioning factor / is shown 
in figure 4.11, where the pair (x, was conditioned to (x°, and 

the nullspace was enforced to be equal to the conditioned vector x°. 



Summary As an important result of the analysis of the error made by the ap- 
proximated representation, we conclude that it is absolutely necessary to con- 
dition an uncertain homogeneous vector and to make sure that the nullspace 
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Fig. 4.11. Plot of relative bias of Euclidean normalization in normal scale (left) 
and logscale (right). To obtain this plot, a point x = (u,v)^ was conditioned by 
factor /, obtaining x° and the nullspace of the covariance matrix was forced 

to be x°. 



of the accompanying covariance matrix contains only the homogeneous vec- 
tor itself. If these two conditions are met, the bias in the approximation 
can be neglected for all operations of geometric reasoning, provided the un- 
certainty in the Euclidean parameters is sufficiently small. Additionally, the 
bias is non-negative yielding only too optimistic approximations of the true 
variance. 



4.4 Construction of Entities 

In the last section we have analyzed the errors that are involved in the ap- 
proximate representation of uncertainty. We now present a method for the 
unique construction of geometric entities, for which the uncertainty is repre- 
sented using definition 10. Furthermore we will shortly discuss constructions 
of entities by projective transformations. 



4.4.1 A Statistical Approach to Join and Intersection 

The algorithm in table 4.1 implements statistical versions of the unique con- 
structions introduced in chapter 3. It takes advantage of the first-order error 
propagation and uses conditioning of the homogeneous vectors as well as 
projection of covariance matrices to reduce the bias. 

To guarantee that the proposed algorithms yields proper uncertain geometric 
entities as defined in definition 10 on page 104, we have to prove the following 
proposition: 

Proposition 6 (Uncertain constructions yield proper entities) 

Assume two uncorrelated uncertain geometric entities (x, J7xx) o,nd 
(y, ^yy). The homogeneous part of the vectors is larger than the Euclidean 
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Join and intersection with uncertainty 

Objective: given two uncorrelated uncertain geometric entities (x, X'xx) 

and (y, Syy), compute the uncertain join resp. intersection 

(z, S zz) ~ (x, .^xx) A (y , Syy') TCSp. (z, Szz) ~ (x, .^xx) n (y, .5^yy). 

1. If necessary, condition x and y with a factor / as determined in 
algorithm 2.1 using fmin = 0.1. Scale the Euclidean parts of the 
covariance matrices in the same way. We obtain (x°,X’xOx“) and 

(y 

2. Change the nullspaces of X’x°x“ and Sy°y° to {x} resp. {y} using 
orthogonal projection Px and Vy, cf. (4.18) on page 111. 

3. Compute the new conditioned uncertain geometric entity 

(z°, ^.o,o) = (U(x°)y°, U(x'’)^yyU^(x°)+V(y°)SxxV^(y°)) 
where the Jacobians U(x°) and V(y°) are chosen according to ta- 
ble 3.1. 

4. If necessary, recondition (z°, X’z°z°) by scaling its Euclidean 
part by 1// to obtain the constructed uncertain geometric entity 
(Z, Szz)- 



Algorithm 4.1: Algorithm for computing join and intersection of uncertain geo- 
metric entities. The conditioning of the coordinates and the change of the nullspace 
of the covariance matrices reduce the bias in the resulting uncertainty. 



part and the nullspaces of the covariance matrices are Af(Sxx) = x and 
JV{Syy) = y. Then for the join n or the intersection A the pair 

(z, Szz) = (U(x)y, U(x)i:yyUT(x) -f V(y)i:xxVT(y)) (4.50) 

with U(x) and V(y) from table 3.1 is a proper uncertain geometric entity 
according to definition 10. 

Proof: We have to show that A/’(X'zz) H £(x o y) = {0} with o being the 
join or intersection operator and f (x) = {x}^ for points and hyperplanes in 
2D and 3D resp. £(L) = for 3D lines. From proposition 1 on the 

following page, we know the nullspaces of the addends of 
Szz = the remainder of this proof, we will only cover the 

join-operations, the intersection operation is skipped due to duality reasons. 

- Case 1: 1 = x A y = S(x)y = — S(y)x 

As A/’(X'i\) C {x} and AZ’(X'n) C {y}, then because of proposition B.2 
A/'(il'n) C { 1 }^ if x y. If they are identical, the line 1 is undefined and 
rk(i:n) = 2. 

- Case 2: L = X A Y = TT(X)X = -TT(Y)X _ _ 

Because of propositions B.2 and 1, Sll) c (x(Tr(x))nx(Tr(Y)))u 
{L}. The image X(TT(X)) = X(CTT(X)) contains the duals of all lines 
incident to the point X. As the same applies for Y, the nullspace A/’(X'll) 
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contains the dual of all lines L incident to both X and Y. If X 7 ^ Y, 
then Af(X*LL) = {L, L} and thus the assertion Af( XLL)n{L,L}^ = {0} 
holds. If X = Y, the line is undefined and the nullspace of Xll is at least 
three-dimensional, thus rk(X'LL) > 3. 

- Case 3: A = X A L = Tf^(X)L = f^(L)X 

Because of propositions B.2 and 1, A/’(X'aa) C ^{X} nX(r^(L))^ U {A}. 
Note that X is interpreted as a plane with the homogeneous coordinates 
of the point X. Taking the dual, we obtain {X} n X(r (L)), which is 
zero if X ^ L, as r^(L) contains the canonical planes of the line L; thus 
-S'aa C {A}. IfX G L, then the plane A is undefined and rk(XAA) > 3. 
□ 

To prove the last proposition, we needed the following statement: 

Lemma 1 Assume two uncorrelated uncertain geometric entities (x, Xxx) 
and (y, Xyy). The homogeneous part of the vectors is larger than the Eu- 
clidean part and the nullspaces of the covariance matrices are JV{Sxx) = x 
and Af(X'yy) = y. Let z = U(x)y be a construction as in table 3.1. Then 

Af(U(x)X'yyU'r(x)) C Af(U'^(x)) U {z} (4.51) 

Proof: The covariance matrix = U(x)X'yy U^(x) can be decomposed into 
-S'zz = U(x)HH^U^(x) with H being a n x (n — 1) matrix and nullspace 
A/’(H^) = JV{Syy) = {y}. It is sufficient to consider H^U^(x) only. 

It is obvious that the nullspace A/”(U^(x)) can be part of A/”(H^U^(x)). On 
the other hand, if there is a vector t such that y = U^(x)t, then this vector is 
also an eigenvector of H^U^(x). We can show for each operation, that t = z, 
for duality reasons we only consider the join operations: 

- l=xAy 

In this case we consider m = S^(x)t = — S(x)y with t being interpreted as 
a 2D point. We need a t, such that y = m, since y and y are represented 
by the same vectors. Thus a necessary condition for the existence of t 
is that X G y x^y = 0. Taking the dual of the construction, (y = 
— S(x)t) o— • (— S^(x)y = t), it follows that t = 1 . 

- L=XA Y _ 

Here U^(X) = TT^(X) and we consider B = TT^(X)T, where T is a 3D 
line. We need a T, such that Y = B and a necessary condition for the 
existence of T is that X G Y AA X^Y = 0. Again, (Y = — TT^(X)T) o— • 
(TT'^(X)Y = T) and thus T = L. 

-A=XAL _ _ 

(i) For UT(X) = TT(X), we consider M = TT(X)T or M = TT(X)^ 
where T is considered being a 3D point. We need a T such that L = M 
or equivalently L = M. A necessary condition for the existence of T is 
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X G L. From (L = TT(X)T) o-. (TT^(X)L = T) it follows that T = A. 
(ii) For U^(X) = f(L), we consider B = f(L)T, where T is interpreted as 
a 3D point. We need a T such that X = B, thus a necessary condition for 
the existence of T is L G X. From (X = r(L)T) o— • (F (L)X = T), it 
follows that T = A. □ 



4.4.2 Validation of the Statistical Approach 



We validate that the algorithm by Monte Carlo tests in 2D and 3D where 
all geometric entities are involved. We then compare the sample mean and 
sample covariance with the computation using first order error propagation 
as in equation (4.23) on page 111. 

In both tests we generate a sufficiently large number of 2D resp. 3D points, 
where each point is constructed by some join and intersection operations. 
Liu et al. 1996 suggest five hypotheses tests for testing multivariate data: 
the randomly generated data is assumed to be Gaussian with unknown mean 
/i. and covariance X and it may be tested against the approximated mean p 
and covariance S. For our purpose, we choose the following tests: 



Hp, = (i with S being untested. 

This test decides whether the generated point data has the computed mean 
fi using bilinear constructions. The computed covariance is not used. 

= S with p being untested. 

This test questions whether the covariance of the generated point cloud 
has the approximated covariance using first order error propagation. The 
mean p of the point cloud is disregarded. 

Hp = p and S = S. 

The last test is a combination of the above, simultaneously testing covari- 
ance and mean of the generated point cloud. 



Additionally, we define two measures reflecting the relative error of between 
the sample mean y and covariance Syy on one side and the approximated p 
and covariance X on the other side. 



rcE 



rev 



trace(X) 



S- I! 



(4.52) 

(4.53) 



The number of iterations for the Monte Carlo simulations were fixed to 
lOO’OOO for all tests in 2D and 3D. 

For 2D, we reconstruct a point by the intersection of two lines, which in turn 
were each constructed by joining two random points, see table 4.2. Thus both 
join and intersection operation were used. The three hypothesis tests have 
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Monte Carlo test for the construction a 2D point 

Objective: Test the validity of the algorithm 4.1 by repeatedly 

constructing a 2D point out of 4 random points. 

1. Repeat N times 

a) generate 4 random points Xi according to a Gaussian dis- 
tribution with (Ja;, ay and a^y and and transfer them 
to homogeneous vectors Xi. 

b) compute 2 lines li = xi A X2 and I2 = xs A X4 

c) compute point y = li n I2 

d) normalize by {y, 1)^ = No(y) 

2. Gompute sample mean y and sample covariance Syy 

3. compute point (y, Syy) as in step 1 using the algorithm in 
table 4.1. 

4. compare sample statistics {y,Syy) with computed statistics 
(l/i ^yy) 



Algorithm 4.2: Monte Garlo test for the construction of a 2D point. 



been accepted up to an input noise of 10%, see table 4.2. Only for 25% and 
50% input error the approximated covariance matrix was not accepted to be 
equal to the sample covariance matrix. Similarly, the relative errors of mean 
and covariance matrix is less than 10% for small noise levels. 



Table 4.2. Results of 2D Monte Garlo tests: testing the approximated covariance 
with the sample covariance matrix according to table 4.2. lOO’OOO samples have 
been generated for each error level. 



Test-type 


1 % error 


10% error 


25% error 


50% error 


Test 1 

y, = yo with unknown E 


0 (+) 


0 (+) 


0 (+) 


0 (+) 


Test 2 

E = Eo with unknown y 


3.25 (-k) 


4.59 (-k) 


43.94 (-) 


100855.08 (-) 


Test 3 

E = Eq and y = yo 


5.24 (-k) 


6.52 (-k) 


45.65 (-) 


100903.15 (-) 


relative error cov 
ll^-^o||/||r|| 


0.05 (-k) 


0.06 (-k) 


0.21 (-) 


0.99 (-) 


relative error mean 
IIm - yo\\/^Jtva.ce{E) 


0.04 (-k) 


0.04 (-k) 


0.04 {+) 


0.14 (-) 



For 3D, we reconstruct a 3D point given three planes which were constructed 
by random 3D points. 3D lines were used as intermediate entities and again 
both join and intersection were used, cf. table 4.3. In this case, the tests have 
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Fig. 4.12. Results of 2D Monte Carlo test using varying a. Depicted are the points 
that have been constructed according to the algorithm in table 4.2. The top two 
rows show one geometric construction with varying input error. In the last row, two 
other geometric situations are depicted. On the lower right example one sees that 
if the intersection is not well-defined and the input error becomes larger, the error 
distribution is not Gaussian anymore. 



Table 4.3. Results of 3D Monte Carlo tests: testing the approximated covariance 
with the sample covariance matrix according to table 4.3. lOO’OOO samples have 
been generated for each error level. 



Test-type 


1 % error 


5% error 


10% error 


25% error 


Test 1 

/r = /To with unknown S 


1.41 (+) 


1.33 (-k) 


2.53 (-k) 


1261.33 (-) 


Test 2 

E = Eq with unknown /r 


4.35 (-k) 


34.76 (-) 


43.94 (-) 


978255.37 (-) 


Test 3 

E = Eq and = fio 


5.78 (-k) 


36.9 (-) 


45.65 (-) 


979500.98 (-) 


relative error cov 
ll^-^o||/||5|| 


0.06 {+) 


0.08 (-k) 


0.16 (-) 


1.00 (-) 


relative error mean 
IIm - tto||/Vtrace(r) 


0.04 (-k) 


0.03 (-k) 


0.04 (-k) 


0.98 (-) 
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Monte Carlo test for the construction a 3 D point 

Objective: Test the validity of the algorithm 4.1 by repeatedly 

constructing a 3D point out of 9 random points. 

1. Repeat N times 

a) generate 9 random points Xi according to a Gaussian dis- 
tribution with a full rank 3x3 covariance matrix SxiXi 
and mean Hx ■ 

b) compute 3 lines Li = Xi A X2, L2 = X4 A X5, and L2 = 
X7 AXg. 

c) compute 3 planes Ai = Li A X3, A2 = L2 A Xe and 
A3 = L3 A Xg 

d) compute line M = Ai n A 2 

e) compute point Y = M n As 

f) normalize by (Y, l)"^ = No(Y) 

2. Gompute sample mean Y and sample covariance Syy 

3. compute point {Y,Syy) as in step 1 using the algorithm in 
table 4.1. 

4. compare sample statistics (Y, Xyy) with computed statistics 
{Y,Syy) 



Algorithm 4.3: Monte Garlo test for the construction of a 3D point. 



been accepted for 1% and 5% input noise. For 10% input noise, the tests 
involving the covariance matrix were rejected, though the test- values remain 
relatively small and the relative error of the covariance matrix was only 8%, 
see table 4.3. 



4.4.3 Construction Using Geometric Transformations 

Given a projective transformation H with its covariance matrix J7hh of 
h = vec(H^), the uncertain entity (x, Sxx) is transformed to a new en- 
tity (x', X'x'x') with x' = Hx. The computation of Sx'x' can be done using 
error propagation 



Sx'x' = Hi:xxH^ -b {In G x'^)i:hh(/n G x) (4.54) 

where In is an identity matrix of dimension n, which is equal to the number 
of rows of H. 

It remains to be shown that the resulting entity is a proper uncertain entity 
as defined in definition 10 on page 104: as a prerequisite we assume that both 
the transformation and the given entity are proper uncertain entities. In this 
work we only prove the necessary property of for homographies in 2D 

and 3D. For the transformations involving a projective camera, including the 
fundamental matrix, the proofs are similar but slightly more complicated. 
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Given a homography H, we consider the first addend in the equation (4.54), 
= H J7xxH^. As the homography is assumed to be regular, the possible 
nullspace depends only on i7xx- As we assume = x, one 

obtains = {M'^x}. Because adding a second covariance matrix 

would at most lead to the same nullspace, the condition of in proposition 2 
reduces to 

x''^M^'^x = 0 x"^M^^x' = 0 x"^x = 0 

The proof for a projective matrix P can be done in a similar way; for P^, Q 
and Q one has to take both addends of the error propagation into account. 
The validity of constructions using the fundamental matrix is then given by 
the relation F = Q2Q1 • 



4.5 Testing Geometric Relations 

As we have seen in chapter 3 the check of geometric relations between two 
geometric entities was a simple test whether a distance vector d or D resp. 
distance scalar d was equal to zero. A simple case is the incidence of a 2D line 
1 and a 2D point x with d = l^x,, see figure 4.13(a) In practical applications 
this distance is generally not zero and usually one chooses a small value e 
and test whether d < e. 

The problem is to choose a value e, e.g. for an error-ribbon around the 2D 
line 1, see figure 4.13(b). But even after choosing a fixed e this solution is 
not optimal as it is not dependent on the position of the points and lines 
and their confidence regions, see figure 4.13(c). The choice of e becomes even 
more problematic when testing 3D entities. 

This section introduce a method to choose appropriate values for e for the 
relations introduced in chapter 3 by exploiting simple hypotheses test tech- 
niques known from statistics. 

4.5.1 Hypothesis Testing 

The question “7s the point x^ incident to the line 1 ?” can be solved by a 
statistical hypothesis test, cf. Papoulis and Pillai 2002: we want to test 
the hypothesis 77q: “T/ie point x^ is incident to 1” against the hypothesis 
Hp. ‘‘‘‘The point x^ is not incident to 1”. For general relations, the measure 
that we want to use is the distance vector d or D, in the 2D incidence 
case it collapses to a scalar d = l^x. As we assume that the homogeneous 
vectors representing the geometric entities are normally distributed, e.g. x ~ 
iV(x, I7xx),l ~ Af(li -^xx), also the distance is normally distributed. 

We can now make use the following theorem from statistical testing theory 
(cf. Koch 1999 sec. 2.7.2 p. 135), which provides a scalar test value with 
known statistical distribution. 
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(a) 2D inci- 
dence test using 
dot-product 




(b) Test using e- 
error ribbon 




(c) Test using con- 
fidence region 



Fig. 4.13. How to tell whether a point x; is incident to a line 1: we use the dot 
product d = l^Xi to tell (a), but a fixed threshold e as in fig. (b) such that d < e is 
not a satisfactory solution: the threshold does not depend on the location and size 
of the confidence regions of the points Xi and 1. 



Proposition 7 (Hypothesis testing) Given a n-vector d with normal dis- 
tribution d ~ Sdd), fkSdd = n. the optimal test statistic for the hy- 

pothesis Hq : d = is given by 

r = (d-/x)Ti:-i(d-/x)~x2 (4 55) 

where Xr denotes the Xn~ distribution with n degrees of freedom. 

To perform the test, we need to fix the probability a that we reject Hq 
although it is actually true — this situation is called type-I error. The proba- 
bility a is usually a small number such as 1% or 5% and is called significance 
level of the test. 

The critical value ch such that P{T > ch\Ho) = a is given by the (l-o)- 
quantile of the distribution: 



■ Xl — Q;n 

Now the hypothesis Hq can be rejected with a significance level of a if 

T = £Sf^d > CH = xLa;n (4.56) 

Sometimes it is convenient to use the test-ratio Tr = — , so that we can 

^l-a;n 

compare test- values of different hypothesis tests. 

It is crucial to note that a successful hypothesis test T < ch does not validate 
that Hq is true, it merely states that there is not enough evidence to reject 
Ho- 

4. 5. 1.1 Testing Geometric Relations We can directly apply the above 
test method to the 18 relations introduced in table 3.5 on page 78: a relation 
i?(x, y) can be assumed to hold if the hypothesis 



4.5 Testing Geometric Relations 131 



Ho-. d = U(x)y = V(y)x = 0 (4.57) 

can not be rejected, see equation (3.79) on page 79. The covariance matrix 
X'dd of d is given by first order error propagation as 

iJdd = U(x)i:yyUT(x) + y{y)S^^y^{y) (4.58) 

In the 2D-incidence case, the last equation reduces to 

CTd = l^-S'xxl + x'^iJiix 

and thus the test-statistic T = (P/cr^. 

In general .57dd may be singular, if d is a n x 1 vector, r is is the degree 
of freedom of the relation R and r < n. The singularity causes a problem, 
because we have to invert the covariance matrix in equation (4.56). But at 
least for projective relations we can guarantee that the rank of Sdd is not 
less than r, as the following proposition shows: 

Proposition 8 (Minimal rank of X'dd) Let (x, (y, -^yy) be uncer- 

tain geometric entities in 2D or 3D, which are conditioned, cf. section 2.4, 
and the nullspaces for J7xx cind Syy are determined by their homogeneous 
vectors x,y. The bilinear form d = U(x)y = V(x)y for a projective relation 
i?(x,y) is defined in table 3.5 on page 78 with r = dof{R) being the degree of 
freedom for i?(x,y). Then the covariance matrix Sdd has rank rk(I3dd) > f. 
For the reduced distance vector dW = UM( x)y = V[’’](x)y, the covariance 
matrix .57d[r]dW is regular for all cases mentioned in equations (3.53), (3.54) 
and (3.55) on page 69 

Proof: For projective relations, there are three tests with one degree of free- 
dom, which are analogous to the case in section 4.3.3 on page 117. The 
other tests mostly stem from bilinear construction forms, which have been 
analyzed in the proof of proposition 6 on page 122 and the resulting covari- 
ance matrices are guaranteed to be at least of rank r. For the special case of 
the identity of 3D lines we use the bilinear form in equation (3.78), where the 
matrices Ai(L)~^ resp. Z\i(M)^ both have the eigenvector E^. But i is chosen 
such that EJM > 0 resp. EJL > 0, thus with proposition B.l the rank of 
i^dd is at least r. 

Given the reduced distance vector of any relation R, the ranks of (LJ[’’l(x))^ 
and (Vl’’! (x)) equals r, if one uses the reduction algorithm 3.1. The 
nullspaces of (UM(x))^ and (VM(x))^ are then zero, thus H'dWdir] is reg- 
ular. □ 

Generally, proposition 8 holds for orthogonality and parallelity relations, too. 
This time only the sub-covariance matrix of the homogeneous entities, which 
are basically directional vectors, have to be projected spherically. 

But as the conditioning of homogeneous vectors can not be applied to direc- 
tional vectors, the variance of d can be zero resp. the covariance matrix of d 
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of can be of rank 1 instead 2, cf. section 4.3.3. These special cases happen 
only if the relation of the two directional vectors is exactly opposite from 
the tested relations: for example two lines are exactly parallel but are tested 
for orthogonality. If the vectors are almost opposite from the tested relation, 
the test-value is obviously very large. So in practice the singular cases can 
be avoided for Euclidean relations by adding a very small amount to the 
covariance matrix, for example: 

-^d'd' = -^dMdid + 0.0001^ trace (.£'(j[r](j[r] ) Ir (4.59) 

Thus the new covariance matrix Sd'd' is guaranteed to be always regular; 
this adds only a very small bias to all tests for Euclidean relations, which is 
negligible as the test-value is very large anyway. 



4.5.2 Properties of the Approximated Test- Value T 

It is of practical importance to check some properties of the approximated 
test-value T{f) = from proposition 8 where / is the condi- 

tioning factor for x and y: 

- first we discuss the bias between T(/) and the ideal test- value Tg similar 
to the analysis in section 4.3.3. 

- second we check if an important property of the ideal test- value Te is 
preserved: is the approximated test- value T(f) monotonously increasing 
when increasing the distance between the tested entities? 

Bias of the Test-value We first discuss the bias of the ideal test value and 
the computed using first order error propagation. Since the vector d from 
equation 4.57 is essentially an entity that was constructed from x and y, the 
hypothesis test can be interpreted as a test whether the constructed entity 
is undefined (null-vector) or not. Thus exactly the same arguments from 
section 4.3.3 can be applied, where the bias of constructions was discussed 
and found to be negligible under well-defined conditions. 

To demonstrate the existence of the bias, we construct an extreme example, 
which usually does not occur in practice: assume two uncertain 2D points 
for (x, I?xx) and (1, i?n) with A/’(i?xx) = x and Af{Syy) = y. The test on 
identity x = y is done using d = S(x)y and 

i:dd = S(y)i:xxS(y)^ + S(x)i:yyS(x)T 

One may also use the reduced construction matrices S[^l(x) and S[^l(y). 

An extreme case of the bias is now given when considering x^y = 0, see 
figure 4.14. Note that in practice this case usually does not happen, since 
x^y >> 0 if X « y. Assuming x^y = 0, the rank of the covariance matrix 
X'dd is less than 2, which can be proven using proposition B.l on page 183 
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and the fact that Af{Sxx) = A/’(S(y))-*-. But J7dd has to be of at least rank 
2, as the test has two degrees of freedom, thus a naive application of the 
hypothesis testing fails in this example. 




Fig. 4.14. Cross section of the situ- 
ation of two projective points x, y G 
P^, where the homogeneous vectors 
are orthogonal to each other, x^y = 
0, the u-coordinate is skipped for vi- 
sualization purposes. The covariance 
matrices have nullspaces A/’(X'xx) = 
X and Af{Syy) — y, thus the respec- 
tive uncertainty is defined only or- 
thogonal to its homogeneous vector. 



The above problem is an extreme example of a bias caused by the approxi- 
mation of the used error propagation. When we apply a conditioning of the 
uncertain homogeneous vector with a subsequent orthogonal projection of 
the covariance matrices with Vx resp. Vy (see algorithm in table 4.4), the 
bias disappears, see also section 4.3.3: 

To obtain the true value of the test value, we consider the Euclidean points 
X = (xi,X2)~^ and y = (j/i,?/ 2 )^ and their covariances Sxx and Syy The 
distance d and the covariance matrix Sdd is given by 



d = 




^dd = -I 





with J = 



A 0 -1 
\0 1 0 




(4.60) 

(4.61) 



and we obtain the Euclidean test statistic Te = dJ Sddd. A test proposed 
with homogeneous vectors as in table 4.4 yields a test-statistic T(/) depend- 
ing on the conditioning factor /. Figure 4.15 shows the percentage of the 
bias between Tg and T(/) depending on /: with a factor of / = 0.1, the 
bias is about 1%. If the homogeneous vectors are orthogonal to each other 
(/ = 1), the test-statistic is not defined, if they are close to be orthogonal, the 
test-statistic is very large and is smaller the more collinear the homogeneous 
vectors are. 

As a result of the above example we conclude that the bias can be assumed 
to be less than 1 % if the conditioning factor / = 0.1 is applied. If no con- 
ditioning of the geometric entities is used, the bias can be uncontrollably 
large. 
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2D points: (-1,-1) and (0.5, 0.5) 




Fig. 4.15. Bias when testing identity 
of two 2D points x = (1, 1)^ and y — 
— 1(1, 1)^ using algorithm in table 4.4. 
The conditioning factor / varies be- 
tween 0.1 and 3. With a factor of / = 
0.1, the bias is about 1%. The bias is 
maximum with a factor of 1 as the ho- 
mogeneous vectors x = (1, 1, 1)^ and 
y = (—1/2, —1/2, 1)^ are orthogonal 
to each other. Note the logarithmic 
scale of the percentage. 



Monotony of the Approximated Test-value Although we have identified only 
a small bias between the Euclidean test-statistic Te and the approximated 
test-statistic T{f), it may still be possible that T{f) is not monotonically 
increasing dependent on the relative positions of the two entities, which would 
hinder control or guidance of searching algorithms based on T{f). We will 
discuss the monotony property within P^, cf. section 4.3.3. 

Assume a fixed point x = (u,u)^ = (1,1)^ and a hyperplane 1 = (a, = 

(1,6)^, which solely depends on b. Without loss of generality, we assume 
b > 0. The covariance matrices Sxx and Syy are chosen arbitrarily according 
to definition 10. We now want to test whether the incidence relation xH 1 yf 0 
holds, thus X 1 = 0. First, we compute the test statistic T only with error 
propagation, without prior change of nullspace or conditioning: 



{al -\-2aab + CTb^) + (cr^ -b 2 6cr„„ -b b'^a^) 

To check whether T is monotonically increasing dependent on b, we compute 
the first derivative: 

^ + 2 + gfa ) + gg + bauv) _ 2 Cg -b gg + bauv 

((erg -b 2ggb -b gg) -b (gg + 2fogu„ -b 62cr2))2 (cg-bCb)2 

The sums Cg and Cf, are positive definite forms and thus always positive. To 
check, whether Tf, > 0 for all b, it is sufficient to check 

gg > -bauv (4-62) 

If we assume that the nullspace of is x and the nullspace of Syy is y, 
then with equation (4.37) on page 118 the condition (4.62) becomes 

> buv and thus in our case 1>6 (4.63) 

So generally for 6 > 1, we can not guarantee the monotony property. 

But if we apply the conditioning algorithm 2.1, to x and y, then x° = 
{fminlb, 1)''' and y° = {l,fminV and condition (4.63) becomes 1 > fmin, 
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which is true by definition. Note that condition (4.62) does not change if we 
allow any v instead of only v = 1. 

Thus also for ensuring the monotony property of the approximated test- value, 
the conditioning of the entities is crucial. 

Summary As a result of the investigation on the properties of the test-value 
T we again conclude that it is necessary to condition the uncertain entities 
prior to applying any algorithm. In section 4.3, the conditioning was shown 
to be important for construction and normalization. This time, the condi- 
tioning keeps bias of the test-value for testing geometric relations sufficiently 
small; additionally, the test-value is monotonically increasing depending on 
the relative distance between two entities. 



4.5.3 A Statistical Algorithm for Testing Geometric Relations 

Summarizing the section, we propose a generic algorithm for statistically 
testing geometric relations between points, lines and planes in 2D and 3D, 
cf. table 4.4. The algorithm uses the reduced construction matrices see sec- 
tion 3.2.2, which guarantees the covariance matrix to be regular 

provided proper uncertain geometric entities, cf. definition 10 on page 104. 
Thus one can detect non-proper entities within the algorithm if X'(j[r](j[r] is 
found to be non-invertible. 

An alternative algorithm in table 4.5 does not use the reduced matrices but 
rather relies on an existing SVD algorithm to compute the pseudo-inverse of 
i^dd- If an SVD algorithm is available, the latter algorithm may be easier to 
implement but lacks the feature of detecting non-proper entities. 

Opposed to the SVD, an alternative computation of the pseudo-inverse 
uses the expected nullspace Af{Udd) = H of H'dd- Then the pseudo-inverse is 
given by 




as it was proposed in Forstner et al. 2000. This is feasible for most cases, 
but can lead to problems as the nullspace may increase for special cases, 
such as two exactly identical entities that are tested on identity, see proof for 
proposition 6 on page 122. Using a SVD is computationally simpler since we 
know the expected rank of Sdd- 



4.5.4 Validation of Hypothesis Tests 

To validate the algorithm in table 4.4, we generated n = 200 random cubes 
with edge length 1, as in figure 4. 16 (left) and tested 13 relations between its 
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Statistically testing uncertain geometric relations 
(Version 1 with matrix reduction) 

Objective: Given two entities (x, ^xx) and (y,Syy) 

(possibly points, lines or planes) and a relation R G 
{equal, incident, orthogonal, parallel}. Test the hypothesis 
that the relation i?(x, y) holds with a given alpha-value a, where 
r is the degree of freedom for the chosen test, see last column of 
table 3.5. 



1. If necessary, condition x and y as in algorithm 2.1 using 
fmin = 0.1. Scale the Euclidean parts of the covariance matri- 
ces in the same way by first order error-propgation. We obtain 
(x°,X’x°x“) and (y°,SyOyo). 

2. Ghange the nullspaces of X’x°x° and SyOyo to x° resp. y° using 
orthogonal projection Vx.° and Vyo, cf. (4.18) on page 111. 

3. Obtain Jacobians U(x°) and V(y°) according to table 3.5, such 
that U(x°)y° = 0 and V(y°)x° = 0. 

4. Obtain the reduced matrices U^’'^(x°) and V^’'^(y°), see algo- 
rithm 3.1 on page 69. Gompute the distance df*"' = U^'^^(x°) y° 
and its covariance matrix 



^dMdM = uM(x°)i;,o,oUM^(x°)-f vM(y°)i;xOxoVM^(y°) 



•^dl'ldlr] guaranteed to be regular. For Euclidean relations 
(parallelity, orthogonality), one may want to add a infinitesi- 
mally small amount to X'j[r]j[r] to ensure regularity for some 
cases, see equation (4.59). 

6. Gompute chi-square distributed test statistics T = 
dW The hypothesis “x and y fulfill the re- 

lation R" can not be rejected if T < Xi-a-,n- Otherwise the 
hypothesis is rejected. 



Algorithm 4.4: Generic algorithm for testing geometric relations between uncertain 
geometric entities in 2D or 3D. This version uses the reduction if the construction 
matrices to obtain an invertible covariance matrix A'dd of d. Another possibility is 
to use a SVD, see table 4.5. 



entities. Each cube was constructed by first generating 40 random points ac- 
cording to a multi-dimensional Gaussian distribution with a given covariance 
matrix; for simplicity uncorrelated and identical standard deviations cr were 
used for the point coordinates. 

From these 40 random points, 8 were used for the corners, 24 for the construc- 
tions of lines for the 12 edges and 18 for the construction of planes for the 6 
surfaces of the cube. For each of the 13 relations in 3D, a set of hypothesis 
tests was performed: for example for the point-line incidence test, 200 points 
of a specific corner were tested with 200 lines of an incident edge. Some rela- 
tions allowed only 200 • 199 tests as e.g. the test of identity of the same line 



4.5 Testing Geometric Relations 137 



Statistically testing uncertain geometric relations 
(Version 2 with SVD) 

Objective: Given two entities (x, .57xx) and (y,Syy) 

(possibly points, lines or planes) and a relation R € 
{equal, incident, orthogonal, parallel}. Test the hypothesis 
that the relation i?(x, y) holds with a given alpha-value a, where 
r is the degree of freedom for the chosen test, see last column of 
table 3.5. 

1.-3. see algorithm in table 4.4. 

4. Gompute the distance d = U(x°) y° and its covariance matrix 
with the full construction matrices U(x°) and V(y°) 

i;dd = U(x°)^yOyoU^(x°)-f V(y°)i;xOx=V^(y°) 

5. Gompute the pseudo-inverse for the matrix Sdd from 
step 4 using singular value decomposition EDG^: then Sdd ~ 
GD+E^, where the first r diagonal elements of matrix are 
the inverse of the diagonal elements of D, the other n — r are 
set to zero. 

6. Gompute chi-square distributed test statistics T — d^.S')[^d. 
The hypothesis “x and y fulfill the relation R” can not be 
rejected if T < Xi-a-,n- Otherwise the hypothesis is rejected. 



Algorithm 4.5: Generic algorithm for testing geometric relations between uncertain 
geometric entities in 2D or 3D. This is a variant of the algorithm in table 4.4, the 
difference is that it does not use reduced construction matrices to ensure regularity 
of Sdd, but rather compute the pseudo-inverse using an SVD. 



was forbidden. Altogether 518600 tests were performed with a significance 
level of a = 1% and varying cr. 

To validate the usefulness of the hypothesis tests, we compare the number 
of type-I errors with the given a = 1%. The results are summarized in fig- 
ure 4.16(right): here the test-ratio Tr = Tfxl-a-n plotted in order to 
compare test- values between tests of different degrees of freedom. If Tr < 1 
then the hypothesis Hq is accepted, for Tr > 1, Hq is rejected. For a = 0.01, 
99 % of the tests have been accepted, thus yielding 1 % percent of type-I er- 
rors, as expected for a = 1 %. For a = 0.05, only a bit less than 99 % of the 
tests have been accepted. The performance of the test algorithm decreases 
slowly as a increases: more type-I errors occur for a = 0.1 and a = 0.5 which 
was to be expected, since the estimated covariance matrices are generally too 
optimistic, cf. discussion on bias on page 4.3.3. 



4.5.5 Further Improvements 

We shortly discuss further improvements on the testing scheme. 
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Fig. 4.16. (Left) 3D cube and its associated points, lines and planes. For the 
Monte Carlo tests 200 random cubes were generated with edge length 1. (Right) 
Cumulative relative histogram for test-ratios Tr = r/xi-o;n of 518600 hypothesis 
tests, each with varying a and a = 0.01; test-ratios of 1 or less indicate that Hq 
has been accepted, see text for details. 



4. 5. 5.1 Testability Imagine that we want to test whether one of the tips 
of the dome in Cologne is incident to the line which is defined by the East 
Broadway in New York City. We may simplify this problem to a 2D problem 
and assume that the angle of the street is locally determined with a standard 
deviation of 1°. Estimating the distance between Cologne and New York at 
about 6000 km, the displacement error orthogonal to the East Broadway is 
about 100 km when measured in Cologne, cf. equation (C.3) on page 187. 
Assuming a confidence region which is 3 times larger than the standard devi- 
ation, all points in Germany within a strip of about 600 km width are incident 
to the East Broadway in New York City according to the hypothesis test. It 
is easy to see that although the test is accepted for a large number of points 
in Germany, the result is practically meaningless. 

One can argue that the above test failed to reject because of insufficient 
data. To cope with this problem it is possible to check the testability given 
the uncertain data, as suggested by Baarda 1968. The idea is to find a lower 
bound such that testing errors are detectable with a minimum probability, 
say 80%. For more details, see Forstner 1987. 

4. 5. 5. 2 Assumption of Pull-Rank Covariance Matrices For error 
propagation and normalization we assume that the covariance matrices are 
of full-rank with respect to the degrees of freedom, in other words we assume 
that the variance for each Euclidean parameter is non-zero, cf. definition 10 
on page 104. Without this assumption, we run into problems when inverting 
the covariance matrix Sm in equation (4.56). 

If one wants to weaken this assumption, three approaches may be possible: 

- Doing a rank-analysis on covariance matrix .E'dd by computing the eigenval- 
ues and specify a threshold for zero eigenvalues. This has the disadvantage 
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that we have to find a threshold value, which in general is not obvious, 
especially for arbitrarily scaled homogeneous covariance matrices. 

- Specify the additional nullspace in Euclidean space and add it to the rep- 
resentation (x, S-xx) such that we know that an entity is certain in some 
parameters. The subsequent analysis of the error propagation, a possible 
bias and the derivation of the new nullspaces may be quite involving. 

- Replace the zero variances with very small variances compared to the other 
non-zero variances, similar to equation (4.59) on page 132. Assuming that 
the small amount of artificial noise does not influence the final results, we 
can apply the derived algorithms without any restrictions. 

We recommend to implement the third method. All approaches are also valid 
for the next section, which deals with the optimal geometric estimation of 
geometric entities. 



4.6 Optimal Geometric Estimation 

In this last section of the chapter, we will develop a generic algorithm for 
constructing or estimating a geometric entity or a transformation from a col- 
lection of related geometric entities. As for the non-statistical case in chap- 
ter 3, the over-constrained construction of uncertain geometric entities bene- 
fits from the explorations of geometric relations: the construction is based on 
the relations between observations and unknowns, which was outlined in the 
last section. We can use existing techniques such as weighted least-squares 
estimation to provide a generic algorithm for optimal geometric estimation. 
The used estimation does not only find an estimate of the unknown entity, 
but is also capable of correcting the observed entities, cf. also Kanatani 
1996. 



4.6.1 Statistical Model 

We first derive a statistical model for the construction task, where we deduce 
an estimation /3 of a not observable and hence unknown entity (3 from a 
set of observations y^. Here, the tilde denotes that the vector is true but 
unknown, the hat denotes the vector is an estimated entity. As in section 3.4 
on page 80, we assume that the unknown geometric entity (3 is defined by 
a set of observations y(_Ry), which are related to the unknown by a relation 
/3)> holding for the true observations y(fly). 

The statistical model consists of a (a) perturbation model of the observations, 
(b) an implicit model for the observation process and (c) a model for possible 
constraints on observations and unknowns. For simplicity in notations, we 
will only mention 3D entities and their relations explicitly as it is the more 
general case compared to 2D. 
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(a) Perturbation Model of the Observations 

Consider one observed entity yt := y(fly), the vector y is obtained by some 
measurement process and are subject to a perturbation, which we model 
by a true vector y with an additional random noise vector e: 



y^ = yi + ei with N{0,Ee,ei) (4.65) 



For the estimation of geometric entities, two restrictions have to be made 
to eq. (4.65): 

- though being a homogeneous vector, the observed vector y^ has a fixed 
length as it is observed only once. For the sake of simplicity, we may 
assume that the vectors y^ and y^ are of length 1, thus normalized to 
the unit sphere, yi = Ns(yi). 

- as we only allow Euclidean perturbations to the observations, we require 
the pair (y^, .S’yiy,) to be a proper uncertain geometric entity, as in 
definition 10 on page 104. Since in our model -S'yiy, = -S'eien definition 10 
applies to the pair (y, Seiei)- 

The covariance matrix of the noise vector e can be decomposed in two 
factors ^eiei = o’^QeiSi, cf. KoCH 1999, section 3.2.1: a positive definite 
symmetric matrix QsiCi and a variance factor tr^. Qe^ej can be interpreted 
as the initial covariance matrix ^y°y., the inverse can be interpreted 
as a weighting matrix, which might be given in advance by the user. The 
variance factor on the other hand is an unknown variable, which can be es- 
timated. If for example Qe^ei is the covariance matrix which was estimated 
previously, the result of indicates whether the a-priori uncertainties 
of the observations were either too pessimistic (ct < 1) or too optimistic 
(ct > 1). 

{h)Implicit Model for the Observation Process 

The geometric relations R{y(R,i),^) express the connection between ob- 
servations and unknowns, where i?(-, •) stands for incidence, identity, par- 
allelity and orthogonality relation (symbols: €, =, || and T). An algebraic 
expression for such a set of relations was developed in equation (3.84) on 
page 83. We now analyse this expression depending on the type of the 
relations R{y(R^i), f3)- in general, we may have n + m + p + q relations with 



y(^j) = ^ 

y(iiF) II ^ 

y(_L,o -L ^ 



h = 1, • • • , n 

j = I,-- - ,m 

,p 

1 = 1, ■■■ ,q 



for 3D, the possible relations are again summarized in table 4.4. 

Note that y^R^i) is only a placeholder for a set of different entities: the 
type of y(_Ry) depend on the unknown entity: e.g. points are not parallel or 
orthogonal to other entities, but lines an planes may be. Another example 
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Table 4.4. 3D Relations and algebraic expressions between an unknown /3 and 
observations y;. Both /3 and yi may be points, lines or planes. See also table 3.5. 
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is the incidence relation, where an unknown point X can be related to 
observed lines Mg^/j and planes but an unknown line L may have 

incident points Yg^/j, lines Mg^/j and planes Bg^/j. In general, n = ny + 
UM + ns where ny denotes the number of incident points, um the number 
of incident lines and ns the number of incident planes. Likewise, m = 
my + ruM + ms, P = Pm + Pb and q = Qm + Qb- 

Practically, for unknown points X we can have up to three different blocks 
of observations: X = Yj, X G and X G B^. Likewise for unknown lines 
L there exists up to 8 blocks and for unknown planes A up to 7 blocks of 
observations, see also table 4.4. 

We can reformulate the relationships to bilinear expressions, cf. section 3.3 
on page 70 and obtain and implicit algebraic model for the observation 
process 

Wi^,^(y(i^,^),,3) = 0 44> U(y(i^_i))/3 = V(/3)y(fl_i) = 0 (4.66) 

where the matrices U and V are determined by the relation i?, the type of 
the observations y(/{_i) and of the unknowns /3, as defined in table 3.5. 

We can simplify the observation model equation in (4.66) by replacing 
U(y(ij,i)) and V(,3) with the reduced construction matrices L)[’’‘^(y( R.i)) 
and V[’'‘l(;9), where ri is the degree of freedom of relation i?(yi,/3). The 
reduction can be done by the algorithm 3.1 on page 69, where the first step 
of coordinate transformation can safely be skipped as we assume that all 
relations are approximately true. The resulting reduced model 



Wfl.i(y(fl,*)>/3) = 0 U[’'‘l(y(i^,q)/3 = V[’'*l(/3)y(^_,) = 0 (4.67) 
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is equivalent to (4.66) , cf. section 3.3 The redundancy of the model can be 
calculated using table 3.7, as previously for non-statistical case on page 83. 
(c) Model of Constraints on Unknowns and Observations 

The observations and the unknown have to fulfill the homogeneous con- 
straints and possibly the Pliicker condition, We introduce two functions 
h(;9) and g(y) with their Jacobians H and such that 



h(,9) = 0 




|^|2= ^ ^=h3=1 



/3 = H/3 



for non-3D lines. 



for 3D lines 



(4.68) 



g(y(it.0) = 0 44> < 



|y(fl,i)P = yjR^i)y{R,i) = ^{R,i)y{R,i) = i 

for non-3D lines, 

1 r .7 / ^ 

y{R,i) — '^{R,i)y{R,i) — 

M^y{R,^)VJ \0y 

for 3D lines 



(4.69) 



The matrix C denotes the dual operator for 3D lines, see equation (2.31) on 
page 44. The second condition for 3D lines ensures the Pliicker condition 
LJ^Lq = 0, cf. equation (2.20) on page 33. 

Equations (4.65), (4.66) and (4.68) together form a statistical model for the 
observation process using the observed entities and the unknown en- 

tity (3. In general such a model is called GauB-Helmert model, cf. Helmert 
1872, Forstner 2000. A special case is the GauB-Markoff-model, where 
equation (4.66) is replaced by an explicit relation between observations and 
unknowns: y = w*(/3). 



Bloek Strueture of the Model Note that the observation model equation (4.66) 
is actually a set of equations, defined for each block of observations. We could 
stack up all observations in one big vector y, 

y = (ye, y=, yy, y±)"^ with 

yR = (Y(flp), • • • , • • • , 

(4.70) 

As all observations are assumed to be independent from each other, the corre- 
sponding covariance matrix Syy would be a block-diagonal matrix. Similarly, 
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the Jacobians for w(y,/3) would be of diagonal structure, consisting of the 
matrices U(y(fl_i)) and V(/3). 



4.6.2 Iterative Estimation 



Using the previously derived statistical model, we now discuss the problem to 
estimate the best fit of the vectors (3, y and a given an observation vector y, 
a covariance matrix Syy We use a linearized iterative method as described 
in Mikhail and Ackermann 1976, which is covered in more detail in 
appendix C.2. 

^( 0 ) 

We assume that there exists an initial guess for the unknown f3 , the obser- 
vations are given as ~ y(R,i)- The initial guess for the unknown can be 

obtained by using the non-statistical algorithm 3.3 on page 95. The proposed 
solution is an iterative two-step procedure. 

1. we first estimate the vectors using w(y,,9) = 0 and h(/3) = 0 by finding 
the minimum of 

^(y,/3,A,/i) = i(y-y)'^Qj;j^^(y-y)-kA'^w(y,^)-f ^ min 

. . 

where A and /x are Lagrangian multipliers. One can solve this minimiza- 
tion problem by a linearization at y^°^ and \ obtaining the Jacobians 
U(y*-°^) and V(/3^ ^). A solution of the optimization problem with a com- 
plete algorithm can be found in the appendix C.2. If the observations y 
are normally distributed, the solution represents a maximum-likelihood 
estimate. As the problem has a block-structure, the normal equation ma- 
trix of the solution is given by 



N = 



/n hA 

\H oj 



with N = ^ 

R,i 



(4.72) 



After the first step, we obtain an estimate y[fl\)j/9 and ctT), 

2. in a second step, we make sure, that for each y^^^ = fulfills the 

constraint (4.69). This can be done again using an iteration process, where 

the closest vector y to the given y*^^^ is computed, that fulfills g(y^ ^) = 
0. As described in the non-statistical case in section 3.4.3 on page 84, 

x(i) 

we can approximate the constraint g(y ) = 0 locally and therefore 
iteratively apply for 
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^(1) T T 1 ^(0) ^(0) 

y = y - i^'yyG^(Gi:yyG^) ^g(y ) with y := y^^^ 

cf. Kanatani 1996 or Matei and Meer 2000. Here, the matrix G is 
the Jacobian ^ computed at the position y^^\ cf. equation (4.69). It is 

sufficient to use only a couple of iterations, as we expect that y^^^ almost 
fulfills the constraint. 

Note that for the implementation, we recommend to use the actual Jacobians 
derived in the last chapter instead of a numerical differentiation: not only is 
this solutions probably more aesthetical, we also expect a better convergence 
and stability compared to obtaining the Jacobians numerically. A complete 
algorithm can be found in table 4.6. 



4. 6. 2.1 Estimating Projective Transformations So far we have focused 
on unknown geometric entities. Now the task is to estimate an unknown pro- 
jective transformation denoted by M given a set of observed corresponding 
entity pairs (xi,x') with given covariance matrices. Fortunately, the above 
estimation method needs only small adjustments to solve this problem: as de- 
scribed in section 3.5 on page 85, the corresponding entities and the unknown 
transformation form a triple relationship i?(x'; M, x^), for which we know tri- 
linear algebraic expressions, cf. tables 3.8 and 3.9, thus we have explicit forms 
for the Jacobians of the relation. 

To use the iterative estimation scheme, we only have to turn the observed 
pairs of entities (xi,x() into one observation vector yj = (xj,x(^). The 
covariance matrix and Jacobians for w(yi, f3) and g(yi) have to be combined 
accordingly, e.g. for the observation equations we get with m = vec(M) 



Vi(m) 



aw(y^,m) 

dy 



( Ow(yi,m) 

Ox, 




Ow(yi,m) I 
/ 



l/v,(x',m) 0 \ 

2 \ 0 



With these preparations, the algorithm in table 4.6 can be applied for ge- 
ometric transformations, too. One only has to take into account that the 
reduction v!^^(m) of the matrices V.i(m) have to be done separately for the 
underlying sub matrices. 



4.6.3 Generic Algorithm for Optimal Geometric Estimation 

The complete algorithm see table 4.6. It can handle both geometric entities 
such as points, lines and planes in 3D and projective transformations, such as 
a projective camera or homographies in 2D and 3D. Applications of this algo- 
rithm are manifold, see section 3.4 for some examples, that can be optimally 
solved using the algorithm in table 4.6. 
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Statistical optimization of geometric entities 

Objective: find an optimal estimation for an nnknown geometric entity 
Sjjjjj using a set of observed entities j , that are 

supposed to be related to the unknown entity by the geometric relations 
R{y(n^i), (3). Additionally, compute the fitted observations and the 

estimated variance factor a. 

If the unknown /3 is a transformation, then the observed entities are cor- 
responding pairs {y(R,i),y[n,i))- 

1. If necessary, condition the observations y(_R,i) with a conditioning fac- 
tor / as determined in algorithm 2.1 using fmin =0.1. For transfor- 
mations, obtain two different factors /, /'. Scale the Euclidean parts of 
the covariance matrices in the same way. 

2. Change the nullspaces of to y(n^i) using the orthogonal 

projection cf. (4.18) on page 111. 

3. Group the observations in b blocks depending on the type of relation 
R and type of the observation vector y(_R,i). 

For each block i — 1, ■■■ ,b, define a function Ai(y*,/3*), Bi(y*,/3*) 
that return the Jacobians dw{y, (3) /d(3 at a given position (y*,l3*): 

a) Obtain Jacobians U(y*) and V(/3*) according to table 3.5 

b) Select those r rows from U(y*) and which have the largest 

Ll-norm with respect to one of the matrices, see the second step 
in algorithm 3.1 on page 69. One obtains the reduced matrices 
UM(x') and vM(y'). 

Similarly, for each block i, dehne a function G(y*), that returns the 
Jacobian of g(y*) as defined in equation (4.69). 

If the unknown is a transformation, the observation vector consists of 
stacked pairs of homologous vectors, cf. section 4.6.2. 1. 

Lastly, define a function for H(/3*), cf. (4.68). 

^(0) 

4. Compute an approximate solution for /3 using the direct algo- 
rithm 3.3. 

5. Compute the iterative solution of Gaufi-Helmert model for block struc- 
tures as outlined in the algorithm in table C.3 on page 195. 

6. If necessary, recondition the estimated ^/3, and the fitted ob- 

servations by 1//; for a transformation (3 use equation (2.25) with 

fj'- 



Algorithm 4.6: Algorithm for optimally estimating geometric entities using 
weighted least squares. 



Alternative Solutions for Iterative Minimization As described in Forstner 
2001a it is possible to extend the eigenvector solution described in section 3.4 
to take the uncertainty of the observations in to account. The solution is sim- 
ilar to Leedan and Meer 2000 and Matei and Meer 2000 and exploits 
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the fact, that the problem is bilinear in each pair of unknowns and observa- 
tions. One can minimize 



17 = W^X'wwW 



= /3^ I ^ I /3 ^ min 

dR,i) 



under the constraint /3^/3 = 1. An additional constraint such as the Pliicker 
condition has to be imposed to the unknown in a second step. Likewise, any 
constraints on the observations have to be imposed manually. The advantage 
of this estimation is that it is supposed to converge faster than other iterative 
methods, Matei and Meer 2000. 

Another method for iterative estimation is the commonly used Levenberg- 
Marquardt iteration, for example described in Press et al. 1992. Compared 
to the previous methods it adds a regularization to avoid local minima. The 
reason for using an iterative GauB-Helmert model is not only the fact, that it 
is a classical geodesic estimation method. It also fits nicely into the geometric 
framework, since relations between entities are algebraically expressed by 
implicit equations. 



4.6.4 Example: Estimating Entities of a Cube 

We now test the estimation algorithm using an estimation of cube entities by 
a large set of different observations. For example, assume that a 3D line L 
representing one edge of the cube is unknown; the observed entities are: (1) 
incident points Xf , (2) incident lines L7, (3) collinear (i.e. equal) lines Lf, (4) 
parallel lines (5) orthogonal lines L^, (6) incident planes Af , (7) parallel 
planes a| and (8) orthogonal planes A/^. All observed entities are derived 
by the other 9 edges, 8 corners and 6 surfaces of the cube. We obtain the 
implicit observation model w(L; Xf , L^, Lf, , L^Af , a| , A^) = 0 and 
the Jacobians listed in table 3.5. In the same manner we can also construct 
a point representing a cube corner or a plane representing a cube surface. 
For testing purposes we used artificial data to validate the estimation method 
with the ground-truth. We estimated a 3D point, line and plane being part of 
translated and rotated cubes of size 1. For each of the 8 corners, 12 edges and 
6 surfaces of the cube, 5 observations have been generated randomly with a 
Gaussian error of 0.1% and 1%. When taking into account all possible con- 
straints, the redundancy of the estimations are 57 for the unknown point, 176 
for the unknown line and 162 for the unknown plane. We drew 1000 sample 
cubes and then estimated 10000 points, lines and planes. In fig. 4.17, the first 

row shows the cumulative histogram for the estimated for the points X, 
^ ^ —2 
Y and planes A, over all estimates we get an average of cr = 0.989. The 
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second row depicts the scatter diagram for the estimated point Y, projected 
onto the three principal planes. The 10000 covariance matrices obtained from 
the estimation have been averaged and its confidence region has been plotted 
against the confidence region of the sample covariance matrix of the results. 
The estimated covariances are too optimistic compared to the sample co- 
variances, which is conform with the theoretical explorations of the bias in 
section 4.3. 

For the given datasets, all estimations converged within 3 to 4 iterations. The 
stopping criteria for the iteration was defined as follows: the corrections to 
the values of /3 should be less than 1% with respect to its standard deviation. 
The algorithm was implemented in the scripting language Perl and takes 
about 1 [sec] for 50 point resp. plane observations with a factor 1.5 slower for 
lines. 
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Fig. 4.17. Results of Monte Carlo test for point estimations. First row: cumulative 
histograms of the estimated variance factor for the estimated point, line and 
plane. Second row: scatter diagram of the estimated point Y projected on the 
xy xz— , yz— planes. Only 1000 out of the 10000 estimated points are shown. The 
inner confidence ellipse is averaged over all 10000 estimated ellipses, the outer one is 
the sample confidence ellipse based on all estimated points. The empirical confidence 
ellipse is larger by a factor of about 1.2; this validates the theoretical result that 
the estimated covariance matrix is too optimistic compared with the true one, cf. 
section 4.3. 



4.7 SUGR: a Library for Statistical Uncertain Geometric 
Reasoning 

As a final result of this chapter, we compile the introduced algorithms in 
a library called called SUGR, standing for Statistically Uncertain Geometric 
Reasoning. The library is capable of 
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- Representing, combining and estimating uncertain points, lines and planes 
in 2D and 3D 

- Establishing geometric relationships between the entities 

- Representing and estimating projective transformations including projec- 
tive camera transformations. 

For the representation of the entities, we choose the approximate representa- 
tion, defined in definition 10 on page 104. 

Only the following algorithms are necessary to accomplish the mentioned 
tasks: 

- Join and intersection with uncertainty (algorithm 4.1 on page 123) 

- Statistically testing uncertain geometric relations (Version 1) (algo- 
rithm 4.4 on page 136) 

- Statistical optimization of geometric entities (algorithm 4.6 on page 145) 

For testing uncertain geometric relations, both versions using matrix reduc- 
tion or SVD can be used (algorithms 4.4, 4.4). We recommend to imple- 
ment version 1 with matrix reduction, see the discussion in section 4.5.3 on 
page 135. To implement the algorithms, two helper routines are required: 

- Conditioning a set of geometric entities (algorithm 2.1 on page 40) 

- Consistent Reduction of construction matrices (algorithm 3.1 on page 69) 

At the time of this writing, two implementations exist for SUGR, one in Perl 
and one in Java, both can be found at http://www.ipb.uni-bonn.de/ipb/ 
pro j ects/SUGR/ index . html. 

In this chapter, we have shown only some artificial examples using SUGR. A 
practical application is demonstrated in the next chapter in the context of 
Polyhedral Object Reconstruction. 



5 Polyhedral Object Reconstruction 



As mentioned in the first chapter, the automated reconstruction of polyhe- 
dral objects from multiple images is an on-going and challenging task within 
the field of Computer Vision: from a given set of different images of a poly- 
hedral objects and some orientation information about the camera positions, 
the aim is to optimally reconstruct the shape of the polyhedral object. Dif- 
ferent approaches exist, see section 1.5.3, combining various cues from geo- 
metrical, radiometrical and topological observations and possibly exploiting 
pre-defined knowledge about the object. 

The goal of this chapter is not to propose another automatic approach to 
the general reconstruction problem, we rather demonstrate the capabilities 
of the SUGR procedures for statistical geometric reasoning proposed in the last 
chapter, see 4.7. To accomplish this, we almost exclusively exploit geometric 
cues and don’t use any other knowledge about the object other than its poly- 
hedral shape. It is not intended to promote the idea that solely relying on 
geometric cues will solve the reconstruction problem. But we want to demon- 
strate the power of the given geometric constraints of the scene combined 
with a statistical framework. For a successful reconstruction system, it is rec- 
ommended to integrate other cues e.g. coming from radiometry or topology. 
An appropriate approach for the integration of cues could be the application 
of Bayesian networks, cf. Pearl 1988, Sarkar and Boyer 1994. 

To show the capabilities of the SUGR procedures we apply them to reconstruc- 
tion systems with varying degrees of automation, cf. section 1.3.3 on page 7. 
In particular, the two main goals of using statistical geometric reasoning for 
object reconstructions in this work are: 

- pushing the automation limits of user-assisted systems, 

- enhancing the performance characterization of automated methods 

In the beginning of this chapter, we describe the principle workflow of poly- 
hedral object extraction and summarize preliminary steps such as obtaining 
camera orientation parameters and automatic feature extraction from im- 
ages. As an example for user-assisted systems we describe two commercial 
modeling systems and demonstrate how to include the proposed techniques 
of statistical geometric reasoning. To completely eliminate the amount of user 
interaction for object reconstruction, a method for automatically establish- 
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ing corresponding 2D line segments is described, which is completely based 
on geometric cues. As in all automated approaches, there are some deficien- 
cies in this method; the final part of this chapter shows ways to overcome 
these deficiencies by including a minimal amount of user interaction. Com- 
bining techniques of automated systems with minimal and simplified user 
interaction are steps on the way from exhausting manual user input toward 
semi-automatic or even automatic systems as outlined in the first chapter, 
see table 1.1 on page 8. 



5.1 Principle Workflow 

We assume the following principle workflow for a polyhedral object extraction 
system, cf. figure 5.1: given a set of multiple images, one first has to deter- 
mine the camera orientation and possibly automatically extract geometric 
features such as points or line segments from the images. These preliminary 
steps will be covered within the remainder of this section. After these pre- 
liminary steps, one could invoke a user-assisted system to manually extract 
the geometric information that is necessary to reconstruct the shape of the 
pictured polyhedra, see section 5.2. Alternatively, an automatic system may 
replace the manual work of the user, see section 5.3. One may also use auto- 
mated techniques in user-assisted systems, cf. section 5.4. In all approaches 
one can make use of the SUGR procedures. 




User Assisted Systems 




- CyberCity Modeler 

— inJECT 

t 

[ SUGR ] 

Automated Systems 



- Line Matching 




Fig. 5.1. Principle workflow of polyhedral object extraction: from a set of im- 
ages, one first has to determine the camera orientation parameters of each im- 
age (sec. 5.1.2). Additionally one extracts geometric features from the images 
(sec. 5.1.1), such as points and line segments. Then either a user-assisted system 
(sec. 5.2) or an automatic system (sec. 5.3) is used to compute a 3D description 
of the scene. One may also combine interactive and automatic methods (sec. 5.4). 
Both automatic and interactive approaches can make use of the SUGR procedures. 



5.1.1 Feature Extraction 

The first step from digital images to a 3D object is the segmentation of 
the images, i.e. the extraction of image features that are relevant to the 
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subsequent tasks. For the extraction of polyhedral objects, we are especially 
interested in points, straight line segments and eventually image regions that 
correspond to polyhedral surfaces. 

The extraction of geometric features in the images can be done manually by 
simply clicking on a point for point measurements. In case of line measure- 
ments on may click on two points on the image, which are supposed to be 
on the image line. Both points and line measurements require the user to be 
quite precise in her clicks with respect to the orthogonal distance of the line. 
Instead of manual extraction one can also apply image segmentation algo- 
rithms to automatically extract 2D points and line segments. The automat- 
ically extracted features are not necessarily corresponding to 3D object fea- 
tures, but are solely based on the observed image intensities. 

Note that intensities are measurements of the physical real world and con- 
sequently are only precise up to some degree. Obviously the automatically 
extracted geometric features are also uncertain and their precision depend 
on the measurement of the intensities. Therefore the segmentation algorithm 
should not only deliver the geometric entities but additionally some informa- 
tion on how precise these entities are to be expected. 

In this work we used the feature extraction system FEX for image segmenta- 
tion (Forstner 1994, Fuchs 1998). FEX simultaneously extracts a set of 
points, line segments and homogeneous regions from one or multi-channel 
digital images. The extraction process of this system can be divided in 
three steps, cf. figure 5.2: first, the image is completely subdivided into non- 
intersection regions, in which either points, line segments or regions are sup- 
posed to be extracted {ternary image). To do this, each pixel is labelled as 
one of the three feature types according to the gradient of the image intensi- 
ties in its neighborhood. To decide on the feature type a hypotheses testing 
is invoked which is based on a homogeneity or a shape criterion of the image 
intensities. For reliable decisions one needs to estimate the noise of the image 
intensities (Brugelmann and Forstner 1992). 

In a second step, the geometric features are precisely localized within each of 
the regions — again using the intensity gradients within a specified area. Fi- 
nally, in a third step, the neighborhood (or topological) relationships between 
the extracted features is determined using a Voronoi-Diagram. An overview 
of the segmentation process is found in figure 5.2. 

We assume that the feature extraction yields realistic estimates for the uncer- 
tainty of the position of the points and line segments. Otherwise one has to 
calibrate the variance factors, for example based on the residuals of a bundle 
adjustment. 

5.1.2 Acquiring the Camera Parameters 

In the last chapter we have outlined an algorithm for the computation of the 
camera parameters of one image. The determination of camera parameters 
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input-image 




Step 2 I V 

localization 



features 



neighborhood relations 




Fig. 5.2. Overview of the segmentation process of FEX, divided in three steps: 1. 
segmentation of the image in three feature types: points, line-segments and blob 
regions; 2. localization of the features within their region; 3. extraction of the neigh- 
borhood relations between the localized features. 



for a set of images is a central and well-researched topic in Photogrammetry 
(Kraus 1997) as well as in Computer Vision (Hartley and Zisserman 
2000). A large number of different approaches for the computation of camera 
parameters exist for varying situations, but the classical approach of bundle 
adjustment - the joint optimization of camera parameters and observed 3D 
entities - is the most general and commonly recommended solution, for a 
review see Triggs et al. 2000. In classical photogrammetric tasks, bundle 
adjustment is formulated as nonlinear least-squares problem, which is espe- 
cially suitable for problems with no gross outliers. In case outliers exists one 
may pre-label them prior to the optimization or use different cost functions, 
see Huber 1981, Klein and Forstner 1984, Triggs et al. 2000. 

One does not necessarily have to compute the full projective camera matrix 
in order to compute the inherent geometric structure of a scene: Faugeras 
1992 and Hartley et al. 1992 showed that one can use the fundamental 
matrix only to obtain a scene reconstruction up to an unknown homogra- 
phy H. Simply stated, one chooses an arbitrary projective camera matrix 
for the first camera. Then the camera parameters for the other images are 
completely determined by the given fundamental matrices. One can now do 
any geometric reasoning with the assumed projection matrices - but without 
using affine or Euclidean relations. At the very end one can find the homog- 
raphy H to obtain the real Euclidean projective camera matrices (“stratifying 
the reconstruction”); for example, this can be done using control points. 
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In all cases, we assume a camera, which preserves straight lines. If this as- 
sumption is violated, one can apply a rectification prior to the image analysis, 
for example as described in Abraham and Forstner 1997. 



5.2 Enhancing User Assisted Reconstruction Systems 

As a first application, we demonstrate how to use statistical geometric rea- 
soning for interactive reconstruction of generic polyhedra. We start with two 
existing systems for user-assisted reconstruction of man-made objects and 
describe how they may benefit from the techniques for uncertain geometric 
reasoning. We will then explicitly show the usefulness of SUGR for two spe- 
cific tasks: the reconstruction of building edges and the automatic grouping 
of these edges to surface patches. 

5.2.1 Existing User Assisted Systems 

There exists a number of different user assisted reconstruction systems for 
man-made structures, see Baltsavias et al. 2001 for a recent overview. We 
choose two existing systems, especially suited for building reconstruction: 
CyberCity Modeler and in JECT, which both started as research projects 
and are now commercialized, see Cybercity 2002 and Inpho 2002. The 
system CyberCity Modeler, introduced by Grun and Wang 1999, is 
embedded in a classical photogrammetric environment and contains a method 
for fitting planar faces given a set of well-defined point-clouds. The acquisition 
of the 3D data only involves the measurement of points: a single object, which 
is identified by the user, is measured by first clicking the boundary points of 
the object in each image, then the interior points are specified in arbitrary 
sequence. After the user completed the acquisition, the software starts a 
relaxation process to identify planar surfaces from the given set of labelled 
points. In a final step a global least-squares adjustment is performed in order 
to obtain an optimal estimate of the reconstructed object- 
The inject system Gulch et al. 2000, uses a different measurement tech- 
nique: first the user chooses a building primitive from an extensible set of pre- 
generated primitives, such as saddle-roof buildings or simple boxes. Then, the 
position and shape of the chosen primitive is not measured by point clicks, 
but rather by adjusting a projected version of the SD-primitive in the image 
with a mouse. A set of building primitives are then combined to a complex 
building by following a set of CSG (Constructive Solid Geometry) opera- 
tions (Encarnacao et al. 1997). 

Both systems could benefit from the developed procedures for uncertain ge- 
ometric reasoning: 

- Integration of points, lines and planes. One may enhance the computa- 
tion of the optimal geometric shape of a building by simultaneously using 
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points and line measurements in the images, contrary to only use point 
measurements: as explicated in section 3.1.2, the transfer between image 
and object space can be done simultaneously for points and lines by using 
the backward and forward projection with the matrices P and Q, see ta- 
ble 3.2 on page 61. Additionally in 3D, all possible constructions involving 
any number of related geometric entity can be implemented in a statisti- 
cally optimal manner, cf. section 3.4 and 4.6. 

- Uncertainty analysis. Throughout the system process it is important to 
check whether the accuracy specifications are fulfilled for both intermediate 
and final results. Using the SUGR procedures, it is easy to obtain information 
about the uncertainty of reconstructed objects: assume the measurement 
errors in the images and the precision of the camera parameters are known, 
then one can propagate this information directly to the reconstructed en- 
tities, such as corner points or building edges. The obtained covariance 
matrices can be analyzed and compared with the expected error; one may 
show the user the quality of the obtained reconstructions by traffic-light 
indicators (Pregibon 1986, Forstner 1995). 

- Ease of extensibility, for the development of object extraction system it is 
important that it is easy to continuously create extensions to the existing 
system. Due to the consistency of the representation of points, lines, planes 
and their transformation from chapter 3 it is easy to derive new methods 
and still use the same framework. For example, a geometric regulariza- 
tion of a geometric shape may be done under coplanarity, parallelity or 
orthogonality constraints, cf. Grun and Wang 2001. These constraints 
can be selectively imposed on points, lines and planes using simple bilinear 
expressions for the corresponding relations, cf. table 3.5. 

In the following we will demonstrate the above mentioned benefits of SUGR 
with two examples: the manual reconstruction of building edges and the au- 
tomatic grouping of these edges to polyhedral surface patches. Later on in 
section 5.4, an enhancement using automation techniques is proposed. 

It is possible to add the enhancements to existing systems in order to ac- 
quire additional polyhedral structure to the already generated reconstruc- 
tions. This is especially interesting for the inJECT system: its simple user 
interaction of measuring primitives instead of only points is gained by re- 
strictions of the existing primitive models. But a very complicated polyhe- 
dral structure may be hard to model by using a sequence of CSG operations 
on primitives. In this case it may be simpler to specify the polyhedral sur- 
faces directly and glue these parts to a polyhedral shape, which in turn can 
be combined with existing primitives. This type of reconstruction is an ex- 
ample of combining specific, semantic models (set of building primitives) 
with generic models (polyhedron) - a discussion of such a combination can 
be found in Heuel and Kolbe 2001 where two approaches of automatic 
building reconstruction are compared. 
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5.2.2 User Assisted Constructions of Building Edges 

The first example of enhancing user-assisted systems is the task of recon- 
struction of 3D building edges and corners. The geometric situation in 3D is 
depicted in figure 5.3. We will see that both point and line segments in the 
images can be simultaneously used to obtain optimal reconstruction of geo- 
metric primitives. In the following we will concentrate on the reconstructions 
of edges, as it is the more interesting case. 




Fig. 5.3. Geometric situation for reconstruction of building corners and edges 
including observed image points and line segments (left) and the same scene with 
the corresponding viewing rays and planes (nght). 



Assume that the image has been preprocessed by an image segmentation al- 
gorithm providing points and line segments, cf. section 5.1.1. The user now 
specifies a corresponding points and line segments by clicking on the over- 
laid geometric features, see figure 5.5 One obtains a set M. of corresponding 
features for at least two different images. Note that using automatically ex- 
tracted images features has the advantage that (i) only one click is needed 
for specifying line segments and (ii) that the precision in the location of the 
extracted image features is expected to be higher compared to non-guided 
measurements. 




Fig. 5.4. Optimal estimation of a 3D 
line segment using image points and 
image line segments: first, estimate 
the infinite line L, then compute the 
endpoints of the 3D fine segment by 
the most distant image points resp. 
endpoints, see text for details. 
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Now a 3D line is estimated from the corresponding points and line seg- 
ments using the algorithm 4.6. It is advisable to check the validity of the 
estimation by analyzing the estimated variance factor a^. If the a-priori un- 
certainties from the feature extraction FEX are correct, the expected value is 
equal to 1. Assuming ctq = 1, the variance factor is Fisher distributed; for 
the result of our estimation, we allow a factor of up to 2, which basi- 
cally is equivalent to an hypotheses test with a sufficiently large significance 
level. Thus we can alert the user for possible problems such as mismatched 
features, if the estimated variance factor exceeds a value of 2. It is even an 
option to refuse storing the construction if the value has a very high value. 
This feedback is similar to a traffic-light behavior as described in Pregibon 
1986 or Forstner 1995. 

To obtain a 3D line segment S we compute its endpoints (Xs,Xe) which 
lie on the estimated 3D line. This can be done by computing the project- 
ing lines for all points and endpoints of the matched features A4 and collect 
the intersections with the estimated line Lm ■ The two most distant intersec- 
tion points are the endpoints of the new 3D line segment S(A4 ,Lai)i also 
cf. fig. 5.4. Applying the procedure for all visible building edges yields a set 




Fig. 5.5. Interactive reconstruction using points and line segments: the user may 
click on image features corresponding to a building edge and reconstruct the 3D 
edge using the algorithm 4.6. This procedure can be applied to all visible building 
features obtaining 16 building edges in 3D [right). 



of 3D line segments, cf. figure 5.5: for this example we estimated 16 3D-lines 
corresponding to object edges, on average we used 4 point- and 4 line ob- 
servation for each 3D line. The length of the line segments were found to 
be between 2 [m] and 12 [m]. To asses the constructed uncertainty of the 
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line-segments, we checked the error orthogonal to the line directions at the 
endpoints of each line segment, see fig. 5.6. The average standard deviation 
at the line endpoints orthogonal to the 3D line segments were between 0.02 
[m] and 0.16 [m]. 




Fig. 5.6. Left: The uncertainty of a line segment can be expressed by using its 
infinite line. Given the start and endpoints Xs,Xe of the line segments, the uncer- 
tainty of the position of the line is hence only considered in between these points. 
One can use the error Cc on the plane orthogonal to the line direction to get inter- 
pretable measures for the uncertainty of the line. Right: using the 16 reconstructed 
line segments from figure 5.5, this diagram shows a plot of the errors CTc orthogonal 
to the measured lines at the endpoints. 



Note that the 3D line can not be constructed if all 2D line segments 1' lie on 
the same epipolar plane. Then all viewing planes A'(l') are identical and the 
intersection is not defined. This may happen the camera was moved along 
only one axis, and the corresponding object edge is parallel to the moving 
direction. 

Numerical Example We now want to show some numerical examples of 
construction and error propagation on real data, easing the check of a re- 
implementation. Consider one of the images in figure 5.5: we choose one line 
segment Si and two points Xi,X 2 , that are supposed to lie on the same line, 
see figure 5.7(a). The parameters of the features are extracted as follows 
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where the coordinates are in the row-column coordinate system with the 
origin on the top-left corner. The line segment Si is given using the 5-tuple 
from equation 4.11 on page 107 and its length 1. From these parameters 
one obtains the following uncertain geometric entities (see section 4.2.2 on 
page 105): 
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The confidence regions for the line (li, -S'mJ induced by Si and for the points 
are depicted in figure 5.7(b) resp. 5.7(c), scaled by 10 standard deviations. 
One may now construct a line I 2 = xi A X 2 as a join of the two points xi, X 2 . 
Using the algorithm 4.1 on page 123, one obtains 
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Transforming back to a Euclidean interpretation yields 



Vm) = (103.32 [pel] , 115.63 [pel] ),(/?= 15.5° , 

Od = 0.044287 [pel] , cr^ = 2.53°) 



To check whether the line I 2 is identical to the line li, we first condition both 
uncertain entities with / = 7.6 • 10“'* = 7.6e — 4 and obtain 
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We now compute the distance vector d similar to to (3.63) on page 72 with 
the conditioned vectors: 
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-0.947 


-0.269 


= 


0.0036850488 


V -0.319 


0.947 


0 / V 


0.1 / 




V 0.052256559 / 



As the test of identity of 2D lines has two degrees of freedom, we only need 
to check 2 of the three entries in d The reduction algorithm 3.1 on page 69 
deletes the first row of S(l^), thus we obtain 



d[2] = S[2](l°)l° 



/ - 0.10222 0 
V -0.319 0.947 



-0.947 

0 



)( 



-0.963 \ , 

-0.269 = ( 

0.1 / 



0.0036850488 

0.052256559 



) 



For a reliable decision whether the two lines are perform an hypothesis test 
with a = 0.95 according to algorithm 4.4. The test is not rejected and the 
test-value resp test-ratio are 
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Fig. 5.7. Picture (a) shows an aerial 
image with selected geometric features 
extracted by FEX: a line segment si 
and two points xi, X 2 . The conhdence 
regions of these features are shown 
in (b) and (c), scaled by 10 stan- 
dard deviations. The uncertain line 
(b, ^ 1212 ) is constructed by a join of 
(xi, X'xixi) and (x 2 , see (d). 



T = 0.268 



Tr = 0.0450 



as T < Ti _„,2 ~ 5.9. 

Using the identity test of 2D lines and incidence test of a 2D line and a 2D 
point, we can test collinearity of all 2D features that are supposed to belong 
to the same building edge. Doing this for all images, we can compute 3D line 
segments by using a joint estimation as described above. For the indicated 
image features in figure 5.5 one obtains the following 3D line: 



L 







( 1.83118e-05 3.14034e-06 2.18747e-05 
3.14034e-06 5.91207e-06 1.20933e-05 
2.18747e-05 1.20933e-05 3.90813e-05 
0.00223648 0.00421620 0.00862158 

-0.00674255 0.00125156 -0.00431638 
-0.00090315 -0.00170701 -0.00348846 



0.00223648 -0.00674255 -0.00090315 \ 
0.00421620 0.00125156 -0.00170701 \ 
0.00862158 -0.00431638 -0.00348846 1 
3.0073 0.8935 -1.2174 

0.8935 3.5619 -0.3628 / 

-1.2174 -0.3628 0.4929 / 



The length of the underlying 3D line segment is 8.34 [m]. The maximal cross 
section error is maxijc = 5 [cm], the minimal error mintTc = 1.5 [cm]. 



5.2.3 Grouping 3D Line Segments to Surface Patches 

In the previous step 3D line segments were obtained by manually establish- 
ing the correspondences between 2D line segments. Using the testing algo- 
rithm 4.4 it is now possible to group the computed 3D line segments to more 
complex aggregates. 

First, the line segments are aggregated to 3D corners, i.e. corner-points associ- 
ated with two 3D line segments. This means we have to find line segment pairs 
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81,82 in object space O such that the lines L( 8 i),L( 82 ) are incident and 
non-parallel. These tests can be performed with some reasonable significance 
level such as a = 1%. Furthermore we do not want to group line segments, 
that are too far away. If we know the units in object space, we may specify a 
threshold Td for the maximal distance between the two closest endpoints of 
the line segments 81 and 82 . After identifying two such segments, the corner 
point Xc is estimated and the corner is defined by C = (Xc, 8 i, 82 ). The 
resulting grouping algorithm for finding a set C of 3D corners is thus given 
as: 



Grouping 3D line segments 

Objective: Grouping 3D line segments Si £ O based on incidence 
and distance criteria. 

1 for Si,S 2 G O do 

2 Li :=L(Si),L2 :=L(S2) 

3 if Li n L 2 / 0 and not (Li || L 2 ) 

4 then Xc := Estimate-Incident(Li, L 2 ) 

5 skip if distance(Xc, Si, S 2 ) > Td 

6 C :=CU{(Xc, 81 , 82 )} 

7 done 



Algorithm 5.1: Simple pseudo-code algorithm for grouping 3D line segments based 
on incidence and distance criteria. 



This is a first and simple example of how to apply the SUGR-library, cf. sec- 
tion 4.7 on page 147 to automatically obtain new hypotheses about the ob- 
ject. The new corner hypotheses are regarded as the best explanation of the 
observations, thus the underlying inference is essentially an abduction. 
Finally the corners can be grouped to planar surface patches: one can find 
all those corners Ci,Cj, that define identical planes A(Ci) = A(Cj). The 
planar patches are then defined by the convex hull of all identical corners, 
ignoring a possible concave shape of the surface at this stage of reasoning. 

Fig. 5.8. Grouping of 3D line 
segments to planar patches: first, 
pairs of line segments are grouped 
when being coplanar and non- 
parallel, together with their in- 
tersection Xc, they form a cor- 
ner. Several corners C; may be 
grouped to one planar surface 
patch if the underlying planes are 
identical. 
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5.3 Automated Reconstruction 

The enhancements proposed in the previous section are an improvement of 
the quality of user interactions compared to pure point measurements, but 
it still contained a considerable amount of user interaction. We now describe 
an approach to replace the manual establishment of image line correspon- 
dences to an automatic matching system solely based on geometric con- 
straints (Heuel and Forstner 2001). Combining the matching result with 
the reconstruction and grouping, we have automated the following tasks: 

- matching 2D line segments from multiple oriented images 

- optimally reconstructing 3D line segments 

- grouping 3D line segments to corners 

5.3.1 Matching of Corresponding Line Segments 

The matching of image line segments is considerably more complex than the 
matching of image points: first of all, line segments observed by a segmenta- 
tion algorithm are imperfect: the location of the end-points are not reliable; 
additionally, an ideal line segment might be broken into two or more small 
line segments, that are not connected to each other. A second reason for the 
complexity of line matching is due to the epipolar geometry: the search of 
a corresponding point in a second image to the search on the epipolar line, 
thus restricting the search-space from 2D to ID. For lines, there is no strict 
geometric constraint of the location of the line in the other images. 

The now following matching algorithm 5.2, is essentially a heuristic search 
for finding hypotheses of corresponding line segments; these hypotheses are 
constructed and tested using the SUGR procedures. 

5. 3. 1.1 Selection of Segment Pairs In the first step of the algorithm 5.2, 
a pair of image line segments is selected, that are supposed to be caused by 
a building edge. Although the epipolar constraint does not hold for infinite 
lines, we can make use of the epipolar lines of the endpoints of a line segment: 
given a line segment s with start- end endpoint Xs,Xe, the epipolar lines of 
Xs and Xe define a region (called “epipolar beam”) in the other images, 
cf. Zhang 1994 and figure 5.9(1.). Therefore we define a set f(s) for each 
line segment s, which contains all line segments and points being in the 
epipolar beam. For the epipolar beam, the uncertainties of Xg,Xe have a 
small, negligible influence on the result and safely can be ignored. 

5. 3. 1.2 Construction and Verification of 3D Lines (Steps 1 and 2). 

For a pair of 2D lines li,l 2 we compute the 3D line L 12 by intersecting the 
projecting planes A'(li), A'(l 2 ). This apparently causes problems for specific 
configurations: the uncertainty of the constructed 3D line will be very large 
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1. Selection of segment pair 2. Direct constrnction 




4. Collinearity check of matched 
image features 




5. Optimal estimation 




Fig. 5.9. Overview of an algorithm for establishing correspondences between line- 
segments in multiple views: a line segment pair si , S 2 is selected using the epipolar 
beam f (si) (1.). The lines li, I 2 induced by the line segments are used for computing 
a 3D line L 12 (2.). Then line segments and points in not yet used images are found 
by testing the incidence of their back-projections with L 12 (3.). In each of the images 
a collinearity check of the matched features is used to find matching outliers (4). 
Finally, an optimal estimation of the 3D line segment S(M) is computed by using 
all matched image features (5.) 



when the line segments are close to the epipolar line, see figure 5.10 for an 
example. In these cases the chosen pair has to be rejected: a possible criteria 
for large uncertainty is the error sigmac orthogonal to the line direction, cf. 
figure 5.6(left). The error CTc should not exceed a maximal error which 
has the advantage that the maximal error can be expressed in object 
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dimensions. Therefore it is not possible that a 3D edge is considered, where 
all image lines are on one epipolar line. 




Fig. 5.10. Assume a 3D line is constructed by two image line segments, that are 
almost on the same epipolar line, see images on the right. Then the uncertainty of 
the 3D line is quite large, which can be checked by projecting the 3D line segment 
back to the image, the confidence hyperbola (scaled by 3 standard deviations) is 
significantly larger than expected for the images on the left. 



Find Incident Matches (Steps 3 and 4) After constructing a 3D line L 12 , 
the next step involves the search of matching features in the other N — 
2 images. Again we can use the epipolar beam to reduce the number of 
candidates for the match. To test the candidates we can test the incidences 
of the back-projected image features L'(x') and A'(l') with the 3D line L 12 , 
see figure 5.9(3.), again using the SUGR algorithm for testing relations, cf. 
algorithm 4.4. In case of a successful test we may measure the closeness of 
each image feature feature to the 3D line L 12 by using the test-values T of 
the incidence tests. All matched features are collected in a set A4. Note that 
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we formulated the tests in 3D, but it is equivalent to project L 12 into the 
images and test on collinearity in 2D with the feature candidates. 

The set A4 of matched image features is now checked for consistency within 
each image, figure 5.9(4.): all matched 2D points and 2D line segments U 
in one image should be collinear to each other. We pairwise test collinearity 
between the image features G Ij and Ifc = Im- A feature is then removed 
from the set A4 of matches, if 

- the collinearity test was not successful, i.e. x^ ^ Ij and Ik ^ Im, and 

- the feature has a larger test-value T for the incidence with L 12 than its 
counterpart. 

Note that with the second criterion we take advantage of the monotony prop- 
erty of the approximated test values (section 4.5.2): we assume a feature to 
be further away from the line L 12 if the test-value is larger than the one for 
its counterpart. 

Optimal Estimation (Step 5) With the feature-set A4 it is now possible to 
estimate a 3D line Em with all back-projected point and line segment matches 
using the algorithm 4.6. This is done accordingly to the estimation of the user- 
assisted case and we may reject the feature-sets that give a variance factor a 
higher than a suitable threshold, see discussion above. As a final test of the 
estimated line Em, we check again if all matches in M. are still incident to 

Em- 

The last part of the algorithm is the computation of the endpoints (Xs,Xe) 
of the 3D line segment S which lie on the estimated 3D line, again done as 
in the user-assisted case, see figure 5.9(5.). 

It is possible that multiple 3D line segments Sj have been extracted for the 
same true line segment S. Then all Sj should be equivalent to each other, 
which again is tested statistically. We merge all equivalent 3D line segments 
by taking all 2D feature-sets and recompute the 3D line Em'- 
The approach of computing a 3D line segment solely based on geometric 
constraints is summarized in the algorithm 5.2. 



5.3.2 Examples 

The matching algorithm has been tested on an artificial scene and aerial 
imagery. 

Artificial Example The artificial scene (figure 5.11) consisted of cubes with 
edge-length 1 that were stacked upon each other. Eight views of the 3D scenes 
were produced and a bundle adjustment was computed to obtain optimal 
projective camera matrices. Then the images were corrupted by a Gaussian 
noise of 1 [pel] and segmented using the FEX-system. From these features, the 
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Matching and Reconstruction of Line Segments 

Objective: Compute a 3D line segment S from image feature sets 
J-'’ ,i = 1, . . . , N for N images. 

1 for Si e U • • • U do 

2 for S2 G 5(si) do li ~ l(si),l2 l(s2) 

3 // unique construction using first two lines 

4 - A'(li)nA'(l2) 

5 At ;= 0 // initial match-set At 

6 // search projections of L12 in other image 

7 for X, m G f (li) \ (.A(li) U ^■(b)) do 

8 if L'(x) n Li 2 / 0 then x G At 

9 if A'(m) 9 Li 2 then m G At 

10 done 

11 // pairwise collinearity check of matched features 

12 for i ;= 1, . . . , A; X, mi/2 G At n iF' do 

13 if (x ^ mi/2 or not(mi = m2)) 

14 then TVt := TVt \ {x or mi or m2} 

15 done 

16 skip At if : F* n At = 0 

17 // estimate 3D line by incident matches At 

18 (JLm,^ll,o) := Estimate-Incident(At) 

19 skip yVt if (T > Tct // check estimation 

20 if (Vx, m G At : A'(m) G Lai and L'(x) n Lai 7^ 0) 

21 then compute 3D line segment S(7Vt,LAr) 

22 done 

23 done 



Algorithm 5.2: Pseudo-code algorithm to compute a 3D line segment S from fea- 
ture sets = 1, . . . , A for A images. A feature set consists of image line segments 
si, 2 , mi , 2 and points x. The set f (s) is the set for a line segment s, which contains 
all line segments and points being in the epipolar beam of s. The threshold To- is 
chosen according to an hypothesis test, see text for details. All tests and construc- 
tions are done using the SUGR-algorithms, cf. section 4.7 on page 147. 



matching algorithm 5.2 was applied. It was required that matches should exist 
in at least 6 images. Only true line matches are found, but the endpoints of the 
line segments are sometimes extended due to accidental matches in special 
viewpoints. Additionally some line segments have not been found because 
either they were not visible in 6 or more images or the FEX has not found an 
edge due to low contrast. 

Aerial Imagery The algorithm has been applied to aerial images, where the 
projection matrices had been computed by bundle adjustment. The datasets 
we used were made available within the context of a workshop on automatic 
extraction of man-made objects, cf. Grun et al. 1995and consisted of four 
overlapping aerial images. By selecting image patches with one small-sized 
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Fig. 5.11. Six out of eight images of an artificial seen {top two rows) and the re- 
constructed 3D line segments projected in the images {bottom rows). Only true line 
matches are found, but the endpoints of the line segments are sometimes extended 
due to accidental matches in special viewpoints. Additionally some line segments 
have not been found because either they were not visible in 6 or more images or 
the feature extraction program FEX has not found an edge due to low contrast. 
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Fig. 5.12. first row: extracted line segments and points superimposed on three out 
of four images; second row: projected 3D line segments computed with the proposed 
algorithm using line segments and points. 



building, the algorithm was able to reconstruct most of the 3D lines, see 
figure 5.12, while some false matches were detected and some edges were 
missing. Since the purpose here is to demonstrate the feasibility of SUGR, we 
don’t provide an extensive examination of the building reconstruction results. 
The next section contains a discussion of the properties of the results. 



5.3.3 Discussion 

The sketched algorithm is a first proposal for the extensive use of SUGR for 
matching algorithm. The overall strategy was to build up a set of hypotheses 
and filter out those hypotheses, which do not fit to the geometric constraints. 
To reliably use these geometric constraints, the algorithms from the last chap- 
ter have been extensively used. There are some interesting properties of the 
approach: 

- There are no parameters to be set other than significance levels for the 
hypotheses tests, which was fixed to 95% and a distance threshold Td for 
the grouping of 3D line segments to corners. 

- The step of optimal reconstruction of a 3D line segments with respect to 
the observations is an essential part of the matching algorithm, as it can 
reject bad hypotheses based on the result of the joint estimation. 

~ Image points can be integrated into the matching and may contribute to the 
final 3D line segment. After a pair of 2D line segments have been chosen, 
back-projected points in the other images can be tested for incidence to 
the match hypothesis. One may even allow the observation of two points 
in an image, which can replace a line observation. 
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However the integration of image points may have problems if too many of 
the extracted image points do not correspond to polyhedral corners. Then 
the consideration of points can yield to more false-positives matches: the 
accidental situation that a false image edge is consistent with one or two 
false image points becomes much more likely. 

- A sufficiently large number of images is crucial for a successful reconstruc- 
tion. Using only two images can not work as we can not check the recon- 
structed 3D line. Theoretically three images are enough, as the third image 
can verify the reconstructed 3D line obtained from the first two images. 
But if the verified match is false-positive, there is no way to locate the 
erroneous feature. Only when using four or more images it is possible to 
locate errors in the match. 

Practically one can observe that the geometric constraints are only suffi- 
cient when having four or more images. Using only 3 images, the chance of 
accidental matches is quite high, depending on the scene. But this chance 
reduces significantly by an increasing number of images. 

More than three images are also required when two image lines in different 
images lie on the same epipolar plane as in figure 5.10. Then all lines in a 
third image could be a potential match and one needs a fourth image to 
check the validity of the matching hypotheses. 

- The success of the algorithm depends on the precision of the involved 
entities: if the uncertainty of the image features or the uncertainty of the 
projective parameters become large, more hypotheses will be accepted and 
the search space becomes larger. This is a natural relationship between the 
observations and the unknowns: the less we know about the observations, 
the less we can infer about the unknown. 

“ The algorithm was applied using known camera matrices; but since all 
relations that are used are invariant under projective transformation. Only 
the grouping of corners required a test on parallelity. Note that it is possible 
to do the same reasoning using uncalibrated cameras: then we only use 
the relative orientation of the images and from there artificial cameras, as 
outlined in section 5.1.2. 

- The transfer from matches of the first two views to the third one is done 
via 3D by computing the associated 3D line. One may also perform the 
transfer of the matches only in 2D using the trifocal tensor, cf. Shashua 
AND Werman 1995, Hartley 1995a. 

As mentioned above, the algorithm 5.2 solely depends on geometric con- 
straints and first of all serves as a test-case for the validity of the SUGR 
algorithms. It cannot resolve all issues of the matching problem since other 
sources of information such as radiometry or topology are not used. In detail, 
the inherent problems of the matching algorithm are as follows: 
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- Lack of notion for proximity, the search for matching features given a 
hypothesized 3D line segment is done by projecting the hypothesis into the 
images and then find incident features for this projected line. This criteria 
does not take into account any kind of proximity criteria. Additionally 
the probability that a feature is incident to the hypothesis becomes larger 
the further the feature is away from the point of gravity of the line. Thus 
the matching algorithm can only be applied to small image patches so 
that the proximity condition can be neglected. A possible remedy is to use 
neighborhood relationships between image features, see below. 

- Lack of notion for radiometry: another important problem is that the 
matching algorithm ignores any image intensities that are on the left or 
on the right side of an image edge. This constraint may also filter some 
of the false matches. Note however that even radiometric constraints can- 
not solve all problems: consider repetitive structures such as a crosswalk 
with parallel white stripes (figure 5.13): even a matching algorithm taking 
a local neighborhood image intensities into account could not distinguish 
between the second and the fourth stripe. 

- Computational Complexity: Not taking the epipolar beam constraint into 
account, the above algorithm has a complexity of 0{m ■ n^), where n is 
the number of image line segments and m is the average number of image 
features in iV — 2 images. In worst case the epipolar beam does not have 
an effect on the complexity. 

On our prototype system, implemented with language Perl, a pass of one 
pair of line segment (lines 3-21 in algorithm 5.2) takes on average 5 seconds, 
using the epipolar beam. This number may vary depending on the geometry 
of the scene. While the speed of the system can be greatly reduced with 
an optimized, the complexity issue still has to be resolved by introducing 
more constraints to the solution. 




Fig. 5.13. In some situation, there might arise problems when matching line seg- 
ments because of parallel lines: the pedestrian crossing pictured in the image allows 
for multiple matches, even when considering left and right image intensities. 



5.3.4 Effect of Topological Selection to Matching 

As pointed out in the previous discussion, using only geometric cues such 
as incidence and identity does not incorporate any proximity criterion. To 
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overcome this problem, one may use the neighborhood relationship between 
image features to infer the neighborhood relationship in 3D. An example for 
this approach is described by Heuel et al. 2000 who proposed a system 
for the reconstruction of polyhedra, which uses a strategy for matching 2D 
corners instead of 2D lines, see Lang 1999. The aggregation of basic fea- 
tures to corners does already start in the image domain rather than in the 
3D domain. The early aggregation has the advantage that the subsequent 
matching process becomes simpler. On the other hand, the identification of 
corners in the image is more involving. The result of the corner matching 
reconstruction is then used within a 3D grouping stage, that is similar to 
the one proposed in this work, the result being again polyhedral 3D patches. 
The difference is here that topological constraints are used in additional to 
the geometric constraints: the system infers the neighborhood relationships 
between 3D features from the underlying 2D neighborhoods, which have been 
generated by FEX. The use of topological constraints has the advantage that it 
is computationally much simpler and consists only of a sequence of lookups in 
the 2D neighborhood. Heuel et al. 2000 demonstrated that the application 
of topological criteria prior to geometric tests greatly reduces the number of 
hypotheses for the more involving geometric tests. Furthermore the topology 
selection prevents problems of proximity outlined in the last section. 



5.4 Combining Interaction and Automation 

We now describe two simple methods to combine techniques of automated 
systems with minimal and simplified user interaction in order to overcome 
problems in the automated system. The previous section has demonstrated an 
automated approach for matching 2D line-segments across multiple images. 
One of the disadvantages of this approach is the size of the search-space: first, 
all line segments in the images have to be considered as a possible image of 
a polyhedron; second, for each selected 2D line segments, the only constraint 
for matching features in other images was a weak epipolar constraint. 

To solve the search-space problem, one can consider additional cues, for ex- 
ample from the image intensities. Another practical approach is to lower the 
requirements of automation and allow some minimal user interaction that 
may steer the automated process of reconstruction. In other words, the user 
is supposed to interactively constrain the search space by some simple clicks, 
see also table 1.1 on page 8. 

Choosing a Line Segment One first may enhance the approach described 
in section 5.2.2: there matching image line segments have to be specified in 
all existing images in order to obtain an optimal result. An apparent way 
of using the automated approach is to let the user specify only one line 
segment in one image that refers to a polyhedral edge. Then the outer loop 
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of the algorithm 5.2 can be skipped and likely results in a 3D line segment, 
provided there are enough matching features extracted in the other images. 
If additionally one corresponding second line segment is clicked in a second 
image, the inner loop is skipped too and it becomes almost certain that an 
optimal match is chosen automatically. In cases such as in figure 5.10 this 
reasoning might fail, but the system is aware of this situation and can warn 
the user to choose a different line segment. 




Fig. 5.14. Example of interactively reducing the search-space of the matching 
algorithm. The left four images show the four blobs that have been selected by 
the user by four clicks. The top right images show the projections of the 3D line 
segments found by matching neighboring features of the blob, the bottom right 
shows the set of reconstructed surfaces by clicking on matching surface areas. Note 
that no single corners are shown and the different surfaces are not glued to one 
polyhedral part. 



Choosing Matching Blobs A more effective example of constraining the 
search-space exploits the neighborhood-relationships between the image fea- 
tures that were extracted by FEX. The idea is that the user specifies matching 
areas in each image, which belong to the same polyhedral surface: only line 
segments within this areas should be considered for matching. One can out- 
line the acquisition process as follows, see also figure 5.14: 

- In each image, the user clicks on an area that belong to the same polyhedral 
surface. 

- For each click in each image, the system finds the closest blob, that was 
extracted by FEX. The blob is supposed to correspond to the area that the 
user clicked on. 





172 5 Polyhedral Object Reconstruction 

- All line segments and points that are neighbored to a selected blob are 
collected in a set of image features T’ . 

- Start the matching algorithm 5.2 with the reduced set of features T' . 

- Compute a surface patch from all found 3D line segments, that are coplanar 
to each other. 

The process above only needs user interaction at the first step, namely clicking 
somewhere in an area that refers to a polyhedral surface. The output is at 
best the complete polyhedral surface. This acquisition process is quite an 
advantage to a complete interactive process using point measurement: (i) 
only four clicks were necessary to reconstruct a complete polyhedral surface 
and (ii) no precision is required when clicking on the images, one just has to 
hit the area belonging to the polyhedral surface. 

Of course there are some drawbacks: the approach depends on the quality 
of the image segmentation result - if no image edges were found, the system 
can not reconstruct any edges, which is true for all applications above. Ad- 
ditionally, the extracted image blobs might be merged with other blobs or 
separated from each other with respect to the ideal image blob. Thus one 
may has to click on multiple blobs or take into account that the search space 
becomes larger. An approach for grouping and partitioning of blobs might be 
helpful to resolve this problem, see Heuel and Forstner 1998. 



5.5 Summary 

In this chapter we have demonstrated the feasibility of the SUGR library by 
applying it to the task of polyhedral object reconstruction. It was shown 
that for both user-assisted systems and automated systems, the proposed 
approach of including uncertainty in the reasoning process can be used. The 
statistical approach enabled the reconstruction to take over the precision 
measures from the image segmentation and made it possible to characterize 
the reconstruction results in a consistent manner. 



6 Conclusions 



This chapter summarizes the main aspects of this work, mentions its original 
contributions and finally gives an outlook on possible future work based on 
our approach. 



6.1 Summary 

We have introduced a unified approach for representing, eombining and esti- 
mating points, lines and planes in 2D and 3D taking uncertainty information 
into account. Additionally we have shown how to establish geometric rela- 
tions between entities using a simple hypothesis test. These tools enable us 
to perform geometric reasoning with these entities in a statistical manner. 
The motivation of this work was driven by the task of polyhedral object 
reconstruction. Polyhedral objects can be represented by linear subspaces - 
namely points, lines and planes - which in turn can be treated in a nice 
mathematical way using the Grassmann-Cayley algebra. We have chosen a 
homogeneous vector representation for these entities, thus representing the 
subspaces by projective elements. By subdividing each homogeneous vectors 
into two parts we have established a consistent connection to the underlying 
Euclidean entities. 

Using homogeneous vectors, we have derived simple bilinear expressions for 
the join and intersection operators by using three basic types of matrices: 
the skew-symmetric matrices S(-), r(-) and TT(-). These matrices enable us 
to perform a manipulation of an entity by a simple matrix-vector product 
(table 3.2 on page 61). 

A close look on the construction matrices shows that the rows and columns 
contain special joins and intersections with canonical elements, as suggested 
by the Grassmann-Cayley algebra. This observation can be used to easily 
express incidence and identity relations. Together with parallelity and or- 
thogonality, we obtain a set of simple tests to establish relationships between 
entities, cf. table 3.5 on page 78. 

Using the established relationships between entities, it is possible to formulate 
a generic yet simple estimation scheme for fitting an unknown entity to a 
set of observations, see table 3.3 on page 95. This scheme can also be used 
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to estimate an unknown transformation from a set of observations, which 
is known as general direct linear transformation (DLT). With the proposed 
framework, the idea of a DLT can be reformulated as a problem of establishing 
relationships. This interpretation allows a quick way to establish the algebraic 
expressions for a DLT, which is demonstrated for general homographies and 
the projective cameras. 

For real applications, the proposed algebraic framework has limitations, as it 
takes no uncertainties into account. For the usual case of noisy observations 
we proposed a representation based on homogeneous covariance matrices. 
The choice of covariance matrices constitutes an approximation of the real 
underlying probability density function. It was shown that it is easily possible 
to limit the amount of error that is induced by this choice. The only require- 
ment on the observed data is that the amount of noise should be small, for 
example less than 10%. Based on this approximation, statistical versions of 
the algebraic methods were described, which are combined into a library for 
geometric tasks called SUGR (Statistically Uncertain Geometric Reasoning). 
To demonstrate the feasibility of the SUGR procedures, the tools were applied 
to the task of polyhedral object reconstruction. It was shown that it is easily 
possible to propagate uncertainty information in the images directly to 3D 
reconstructions, which in turn can be used for establishing hypotheses about 
relations in 3D. The tools can be used for user-assisted systems as well as 
automated systems. From the analysis of the automated approach it was 
shown that it is possible to include partial solutions of the automation to 
user-assisted systems. 



6.2 Contribution of Thesis 

The main contribution of the thesis is the development of a methodology 
for doing statistical geometric reasoning on points, lines and planes in 2D 
and 3D. We have provided detailed procedures for the core methods, which 
are generic enough to solve a wide range of geometric problems in Computer 
Vision and Photogrammetry yet always taking the uncertainty into account. 
In detail, the important achievements are: 

- We provided a consistent description of points, lines, planes and their 
transformations in the context of projective geometry. All entities can rep- 
resented as homogeneous vectors including transformations. The homoge- 
neous vectors were subdivided into a Euclidean and a homogeneous part, 
such that the distance of the entity to the origin is equal to the ratio of 
the norm of the parts. 

We translated the join and intersection operators from Grassmann-Cayley 
algebra into a matrix-vector notation. For this we used the already known 
3x3 skew-symmetric matrix S(-) for 2D lines and points and the 4x4 
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Pliicker matrix r(-) for lines; additionally a new matrix called TT(-) was 
introduced, which plays the same role for 3D points and planes as the 
Pliicker matrices for lines and dual lines. The three matrices may also 
serve as a representation for geometric entities or they can be used for 
construction or testing of entities. The interpretation of the matrices in 
table 3.3 on page 66 allow a clarified geometric viewpoint of the structure of 
the algebraic expressions in full analogy to the Grassmann-Cayley algebra. 
With the interpretation of the matrices, it is easy to formulate expres- 
sions for projectively invariant relations such as incidence and identity. 
Additionally, with our choice of a Euclidean and homogeneous part of a 
vector, it is even possible to express projectively non-invariant relations 
such as parallelity or orthogonality. Also distances between entities can be 
expressed in simple formulas, see table 3.5 on page 78. 

Using the projective relations, it is possible to re-interpret the well-known 
DLT method in terms of relations between the corresponding entities and 
the transformation consistently for points, lines and planes. With these 
interpretations, the derivation of algebraic expressions of the DLT is very 
simple, see tables 3.8, 3.9. 

Finally, a generic estimation algorithm of geometric entities and transfor- 
mation by minimizing the algebraic error, similar to the traditional DLT. 
This is an acceptable method for fitting unknown geometric parameters to 
observations as outlined in Hartley 1998. One only has to condition (or 
normalize) the entities to minimize a bias in the solution. As an approx- 
imation of the conditioning operation we proposed to ensure a minimum 
scale difference between the homogeneous and the Euclidean part of the 
vectors. 

- We described how to integrate statistics into the above algebraic frame- 
work: homogeneous covariance matrices have been shown to be sufficient 
to represent and propagate the uncertainty of entities, assuming a reason- 
ably realistic amount of error. Although the representation of the uncer- 
tainty using homogeneous covariance matrices is not unique, one can define 
simple properties to check the validity of an arbitrary covariance matrix 
with respect to its homogeneous vector, cf. section 4.1.2 on page 102. The 
property merely states that a homogeneous vector is unconstrained except 
of homogeneity and a possible Pliicker constraint. For example we do not 
allow points with one or two degrees of freedom, in other words points 
that are only determined on a specific line or plane. Using this property 
a construction algorithm can be formulated for which the first-order error 
propagation is always possible and yields a valid covariance matrix, see 
table 4.1. 

Using the proposed uncertainty representation, it was possible to formu- 
late hypotheses tests for geometrical relations in a consistent manner, cf 
table 4.4. Finally, the generic estimation scheme was extended to exploit 
the uncertainty information using a GauB-Helmert model, where the initial 
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values are computed with the non-statistical version. This provides a sim- 
pler solution to a wide range of geometrical tasks than the one described 
by Kanatani 1996, see table 3.3 

The proposed statistical algorithms have been checked using Monte Carlo 
simulations and enable an efficient way to perform geometric reasoning 
including uncertainty information. 

- As a test-case for the feasibility of the approach, we applied the SUGR- 
algorithms to the task of polyhedral object reconstruction. It was possible 
to integrate the algorithms to systems with varying scales of automation, 
demonstrating that the methods are capable of pushing the automation 
limits of user-assisted systems or enhancing the performance characteristics 
of automated methods. 

Within the automated approach, it was also demonstrated how to build up 
hypotheses of 3D features and aggregates using relation tests and how to 
compute the best estimate given a set of hypotheses. These were the main 
operations for geometric reasoning mentioned in the introduction. 



6.3 Outlook 

The SUGR procedures provide a good basis for further research in the domain 
of geometric reasoning in Computer Vision and Photogrammetry. There are 
a number of paths one may follow to enhance the existing methods or apply 
them to other applications. 

- Constrained geometric entities: as pointed out above, we assume uncon- 
strained geometric entities, thus a 3D point has to be uncertain in all 3 
degrees of freedom. Though a work-around was suggested by adding rela- 
tively small noise to constrained parameters, a theoretically sound solution 
is desirable. Especially because the constraints have to be known to the 
testing and estimation schemes, one could think of adding a rank number 
r to the representation of an uncertain geometric object (x, .S'xx)- This 
would be sufficient when using a singular value decomposition. In general 
though, the theoretical and practical impact of this addition has not been 
covered yet. 

- Correlation between entities: we assumed that a correlation exists only be- 
tween the parameters of one entity, but not between two or more entities. 
It remains to be investigated in which cases the assumption of no cor- 
relation seriously affects the result. Essentially, the error propagation in 
equation (4.23) has to be replaced by (4.22). It seems to be reasonable to 
investigate this issue in the context of constrained vectors. 

- Estimation of aggregates of entities: related to the last point, it would be 
extremely useful to extend the estimation to entity-aggregates, instead of 
only one entity at a time. One has to develop a unified representation of 
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aggregate models such as corners, rectangles, cuboids, etc. using geometric 
entities and a minimal set constraints between them. For example, such a 
method would allow to optimally estimate a 3D cuboid from a set of image 
points and edges. 

Include conics and quadrics: it would be interesting to see how other 
entities such as conics and quadrics could fit into the proposed frame- 
work. Conics and quadrics can be represented by matrices C, for which 
a quadratic constraint for all points on the conic resp. quadric holds: 
x^Cx = 0 (x^ (g) x^)vec(C) = 0. 

Use other cues for matching: with respect to polyhedral object reconstruc- 
tion, immediate extensions are possible: for a successful automated recon- 
struction, especially the matching problem, it is inevitable to use more 
than only geometric knowledge to obtain predictable, robust results. Such 
knowledge could be obtained by assuming only subclasses of a polyhedron, 
or by combining other cues such as image intensities close to the entities, 
which has been successfully demonstrated to be useful by researchers. One 
promising approach would be to extend the example from section 5.4, where 
just the identification of corresponding regions yield a considerable reduc- 
tion of the search-space for finding matching line-segments. The “plane- 
sweep” approach of Baillard and Zisserman 1999 is a good example 
for including image intensities in a similar framework 
Uncalibrated cameras: in our work we assume the projective camera ma- 
trix to be completely known, thus knowing internal and external camera 
parameters. The vision community has been extensively working on cases 
with uncalibrated cameras and shown that a reconstruction is possible up 
to a projective transformation. If one uses only projective relations, all 
tools for statistical geometric reasoning are applicable for the uncalibrated 
case. In practice, it remains to be investigated what impact the assump- 
tion of uncalibrated cameras has for the reasoning algorithms. We expect 
though that the reasoning is not affected if the projective reconstruction 
is close to the Euclidean reconstruction (i.e. the Euclidean infinite plane is 
far away from the origin of the projective reconstruction with respect to 
the scene). 



A Notation 



As stated in section 2.1, we do not distinguish between projective points and 
and its coordinate vectors compared to Faugeras and Luong 2001, p. 78. 
Hartley and Zisserman 2000 take a similar approach, see p.3f. 



x,y,z, a,b,c, 

X,Y,Z, U, U,V,W,T 


coordinates from M 




vectors 
points from 


x',y'] x",y"- £cW,yW 


vectors 

points from the first, second or i-th image 




vectors 
points from 




canonical basis resp. 

points, lines or planes from resp. 


A 


intersection of two projective subspaces, yielding 
a new projective subspace 
constructor yielding new points, lines and 
planes, cf. 3.1 


n 


join of two projective subspaces, yielding a new 
projective subspace 

constructor yielding new points, lines and 
planes, cf. 3.1 


0 


Kroneckerproduct 


x,y 


homogeneous vectors from 
points from 


x',y'; x",y"; xW,yW 


homogeneous vectors from 

points from the first, second or i-th image 
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continued from previous page 


1, m 


homogeneous vectors from 
lines from 




homogeneous vectors from 

points from the first, second or i-th image 


Ns(x) 


spherical normalization of a homogeneous vector 


No(-) 


hluciidean normalization of a homogeneous 
vector, cf. eq. (2.14) on page 26 


s(-) 


construction matrices for points and lines from 

p2 


X,Y,Z 


homogeneous vectors 
points from 


A,B,C,D 


homogeneous vectors 
planes from 


L,M 


homogeneous vectors 
lines from P^ 


L,M 


homogeneous vectors P^^+^ 

dual lines from P^ with respect to L, M 




canonical basis of IR” 

points, lines or planes from P^ resp. P^ 


Tr(.) 


construction matrix for points and planes 


TT(.) 


dual construction matrix: TT(A) = CTT(-) 


r(L) 


construction matrix of a line (Pliickermatrix) 


f(L) 


dual construction matrix r(L) = r(L) 




6x6 matrix for the identity of lines, see 
eq. (3.77) on page 76 


MW 


reduction of a matrix M, cf. section 3.2.2 on 
page 67. 


Xq, Aq, Lq 


euclidean parts of homogeneous point, line and 
plane vectors 


-^hi L}i 


homogeneous parts of homogeneous point, line 
and plane vectors 


H 


3 X 3 or 4 X 4 homogeneous matrix 
homography P^ ^ P^ or P^ — > P^ 


P=(A,B,C)T 


3x4 homogeneous matrix 
projective camera for points P^ — > P^ 


Q= (L,M,N)' 
L = BnC, 
M = cn A, 
N = AnB 


3x6 homogeneous matrix 

projective camera for mapping 3D lines to 2D 

lines 
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continued from previous page 


p = vec(P),q = 
vec(Q), H=vec(H) 


vector representation of transformations P, Q, H. 


Jqp 


Jacobian cf. eq. (3.51) on page 59 


r 


dual or hodge operator for 3D lines, see 


L 


eq. (2.31) on page 44 


F 


fundamental matrix, see eq. (2.23) 


L 


n X n identity matrix 




conditioning matrix lor the conditioning factor 


^u) 


/, eq. (2.25) on page 39 


X(M) 


image or column space of a matrix M 


Af{M) 


nullspace or kernel of a matrix M 




orthogonal projection onto the space with 


'VA,W 


respect to a scalar product < •, • >w- 


X 


random variable 


X 


random homogeneous vector 


Px 


probability density function of sqx 


Mx 


mean value of a random homogeneous vector x 




covariance matrix of a random homogeneous 


■^xx 


vector X 




true unknown entity 


p 


estimation for an unknown entity 


y 


observations 



B Linear Algebra 



B.l Ranks and Nullspaces 

For the analysis of homogeneous covariance matrices in chapter 4 we need 
some statements on the rank of matrix products and the nullspace of matrix 
sums. 

Proposition B.l (Rank of matrix products) Let A be a n x m matrix 
and B a m X r matrix. Then 

rk{AB) < min{rk{A),rk{B)) 

dim{Af{A) f] Image{B))) > max{rk{B) — rk{A),0) (B.l) 

Espeeially, if rk{B) < rk{A) then 

rk{AB) < min{rk{A),rk{B)) dim{Af{A) nA/’((6^)^)) > 0 

Proof: The rank rk(AB) of the matrix multiplication is at most equal 
to min(r/c(/4), rfc(6)). If rk{AB) < mhi{rk{A),rk{B)) or rk{AB) = 
mm{rk{A),rk{B)) depends on the relative position of the nullspace Af{A) 
to the image or column space T{B). Equality holds if and only if A/"(/4) is 
transversal to the image T{B), which is defined as 

dim{M{A) n X(6)) = max(rA:(6) — rk(A), 0) 



One obtains 

rk{AB) < m.in{rk{A) , rk{B)) dim{Af {A)nT(B)) > max( (rfc(6) — rfc(/4), 0) 

The second equation follows from the fact that 2{B) = (Af(( 6^ )-*')) □ 

Proposition B.2 (Nullspace of sum of matrices) Let A and B be two 

symmetric, positive semidefinite n x n matrices. Then 

Af{A+ B) =Af{A)nN{B) 
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Proof: If X G A/”(/4) n Af{B), then /4x + Bx = 0 or (A + 6)x = 0, thus 
X G Af{A + B). To show the other direction of the equivalence, assume x G 
M{A + 6), then x^(/4x + Bx) = 0 which is x^/4x + x^ Bx = 0. Since A and 
B are positive semi-definite, the last equation holds only if x G M{A) and 
xGAf(e). □ 



B.2 Orthogonal Projections 

Orthogonal projections map the vector space K" onto a subspace U or its 
orthogonal complement C/"*" . 

Proposition B.3 (Orthogonal projection) Let A be anxm matrix. The 
orthogonal projection W o/K" onto the space orthogonal to the column 

space A is given by 

<Ta = I - A(A'^A)-^A'^ (B.2) 

The introduced orthogonal projection was introduced according to the scalar 
product x^y, but may also be defined using a weighted scalar product x^ Wy 
with a positive semidefinite n x n matrix 1/1/. 

Proposition B.4 (General orthogonal projection) Let A be a n x m 

matrix and. The orthogonal projection T of M" which maps onto the space 
{A}'*- orthogonal to the column space A according to the scalar product x^ Wy 
is given by 

Ta^w = I - A{A'^WA)-^A'^W (B.3) 

B.3 Kronecker Prodnct and vec(-) Operator 

The Kronecker product and the vec(-)-operator are important to derive tri- 
linear relations between geometric entities and their transformations. 

Definition B.l (Kronecker product) Let A = (oyj be a m x n matrix 
and B = (bij be a p x q matrix. Then the Kronecker product A® B of A and 
B defines a mp x nq matrix 



^ aiiB ■ 






^mn^J 



Definition B.2 (vec() operator) Let A = (a^j be a m x n matrix, then 
vec{A) is a mn x 1 vector, that is formed by stacking the columns of the 
matrix A: 

VGc(A) . — (uii, * * * , ^12? ' ' ' : ^mn) 
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The following important proposition establishes a relationship between the 
vec(-)-operator and the Kronecker product, see e.g. Koch 1999. 

Proposition B.5 Let A be a n x m matrix, B a n x p matrix, C a p x s 
matrix and x an n-veetor. Then 

vec(ABC) = (C^ 0 A)vec{B) = (A^ C^)vec(S^) (B.4) 

Ax = vec{Ax) = {x^ 0 lm)vec{A) (B.5) 

= vec(a:^/4^) = (/„ 0 a;^)vec(/4^) (B.6) 



C Statistics 



C.l Covariance Matrices for 2D Lines 
C.l.l Uncertainty of a 2D Line 

We adopt the representation of Forstner 1992 for the uncertainty of a 
straight line and start with the angle-distance form of a line 

X cos{ip) + y sin{ip) = d (C-1) 

The angle and the distance d are uncertain and with the covariance 
we obtain the covariance matrix of the vector {ip, d)^ 

'"‘"M (C.2) 

\crvd cfl) 

We first assume the simplest case of a line being the x-axis in the 

Euclidean plane with /i(<p) = 90° and /r(d) = 0. We further assume both 
parameters to be uncorrelated, thus aipd = 0. We can reformulate the line 
L in the slope-intercept form y = mx + b, where = p{b) = 0. In 

this case b = d and m = cos(ip) with ab and am being uncorrelated. Then 
the displacement error orthogonal to the line is by simple and rigorous error 
propagation 

CTy = + CTfc (C.3) 

This clearly gives us an hyperbola as a confidence region, cf. figure C.l: the 
line is most precise at the origin with an error of a^ orthogonal to the line. 
The further we move along the line by x, the larger is the displacement error. 
Using first order error propagation, a^ ~ sin{y,{(p)) • cr^ = cr^, thus for this 
special case we can identify ab = ad and a^p = am- The last approximation 
is valid if the a^ < 15°, as dsm(a;) « dx for small x. 

Expressing L(v5,d) in homogeneous coordinates L = (u, w, can be easily 
done some by the following mapping: u = m = cos(f/3), y = sin(</j) and w = d, 
yielding: 
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0 oV 


II 


1 


1 


0 0 0 


V 


VoJ 




b 

o 

o 



(C.4) 



We may now represent any 2D line 1 based on the above representation l^; of 
the X — axis by rotating the line by some angle a and and translating by a 
vector xt = (tx,ty). 



l = T(xt)R(a)U 



( 1 


0 


0^ 




/ 


sin(a) 


cos(a) 


0^ 






0 


1 


0 






- cos(n) 


sin(o!) 


0 




1 


\-tx 


— ty 






1 


0 


0 






lo/ 



The line 1 has now a normal with mean angle = a and a distance 

= tx cos(a) + ty sin(a). The covariance matrix of the line 1 is given 
by 



i:u = T(xt)R(a)i:u.R^(a)TT(xt) (C.5) 

Equation (C.5) yields a rank-2 matrix, where in general all entries are non- 
zero. In particular, this means that we have a correlation between the angle 
and the distance, introduced by the translation. This is depicted in figure C.l. 
Thus another valid representation of an uncertain 2D line is the 5-tuple 

ymj (^’b) 

The point {xm, J/m)^ is the center point or point of gravity of the line, where 
the displacement error orthogonal to the line is minimal. 



C.1.2 Euclidean Interpretation of Homogeneous Covariances 

Assume a line 1 = (a, 6, c)^ is given with its covariance matrix 





/ 2 






< 


^ab ^ac 




^ab 


^bc 




\^ac 


CTbc crl / 



either by construction or by error propagation from some other represen- 
tation, cf. C.1.1. The covariance matrix S\\ may have rank 2 or 3. We are 
interested in the Euclidean interpretation of the homogeneous covariance ma- 
trix is sought as in (C.6). Among others, Clarke 1998 or Faugeras and 
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Fig. C.l. The uncertainty of a 2D line can be represented by a hyperbola (left). 
The point {xm,ymY with smallest uncertainty is called the center point or point 
of gravity of the line. Its position depends on the covariance a^pd (right). 

Luong 2001 suggest to use the duality between the point space and the 
line space (P^)*, where an ellipse in (P^)* yields an hyperbola in P^. 

An alternative method is to use the Euclidean interpretation of 1 = (a, b, c)^ 
and S\\ as outlined in chapter 2 and 4. As proposed in section 2.2.1, a normal- 
ization No(l) yields the angle-distance form (C.l), but we still do not know 
the center point = {xm,ym) of the line. The center point is determined 
by the covariance cr^d or alternatively aac, cfhc which is supposed to be zero 
for the representation (C.6). 

Ad hoc Solution To determine the center point, one has to find a line transla- 
tion T (xm), such that aac = crbc = 0, thus we have to find a unique translation 
such that for the translated line b the covariance matrix 

~ (Xm) 

has zero entries for the elements (1,3) and (2, 3). It turns out that the solution 
is 




and therefore is undefined if cr^cr^ = cr^^, which means that a and b are 
correlated by factor 1. But after a Euclidean normalization, the first two 
coordinates of a line vector refer to the normal of the line and thus a and 
b are indeed completely correlated. Therefore a direct determination of the 
center point fails. 
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Indirect Solution We may reverse the translation and rotation in equa- 
tion (C.5): the rotation is given by the angle the shift u in x-direction 
can be computed similar to Forstner 1992 using the covariance matrix 
and after applying the shift u, the shift in y-direction is given by the distance 
of the transformed line. The complete method for extracting the Euclidean 
representation from equation C.6 from a 3 x 3 line covariance matrix is given 
in algorithm C.l. 



Computation of the Euclidean uncertainty of a 2D line 

Objective-. Compute the Euclidean uncertainty, cf. equation (C.6) 
on page 188 from an uncertain homogeneous 2D line (1, X'n). 

1. Normalize the line 

2. Rotate li such that it is parallel to the x-axis, R is a rotation 
defined by the angle ip of the normal of li : 

3. Determine the shift u of the center of the rotated line 1^^^ 

to the j/-axis parallel to the x -axis. As in Forstner 1992, 
u = where is the (i,j)-th element of 

4. Now determine the shift v along the i/-axis, such that the line 

is equal to the x-axis. The shift v is simply the distance of 
to the origin, thus the third coordinate of 1^^^ . 

= (t(u)I(^), T{v)s[fh{vf) 

5. The covariance matrix has two non-zero entries being crj 
and (T^ as in equation (C.4). The point of gravity is contained 
in the motion H = T(x)T(m)R, the angle p was determined in 
step 2. 



Algorithm C.l: Algorithm for the computation of the Euclidean uncertainty of a 
2D line given a homogeneous covariance matrix 



C.2 Gauss Helmert Estimation 

We will shortly describe the Gauss-Helmert model and its iterative solution as 
it is used in section 4.6. A reference for the GauB-Helmert model is Mikhail 
AND Ackermann 1976, the description here follows the one in Forstner 
2000. We also give an efficient variation of the Gauss-Helmert model, which 
can be applied to the estimation problems discussed in chapter 4. 
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C.2.1 General Gauss Helmert Model 



The Gauss-Helmert-model is assumed to be given by the following entities 
(cf. Mikhail and Ackermann 1976): The W constraints w {^, y ) = 0 
between the unknown parameters (3 and the true observations y are assumed 
to hold. Also the H constraints h{/3) = 0 between the unknown parameters 
alone are given. 

Additionally, we assume G constraints g(y) = 0 only for the observations y. 
First, we will neglect this constraint and impose it later on onto the fitted 
observation y, see also section 4.6.2. 

The true observation vector is related to the observed A^-vector by y = 
y — e, where the errors e are unknown. The observations y are assumed to 
be distributed normally with y ~ N{y,a^Qyy). Finding optimal estimates 
for (3 and y can be done by minimizing 

<P{y,^,X,y) = ^{y - y)'^ Qy^{y - y) + X^g0,y) + (C.7) 



with the IF-vector A and the i7-vector y are Lagrangian multipliers. 

For solving this nonlinear problem in an iterative manner we need approxi- 
mate values f3^ ^ and for the unknowns (3 = (3^ and y = 

. The corrections for the unknowns and the observation are obtained itera- 
tively. With the Jacobians 



A = 



/ dw{(3,y) \ B - f 

\ 9^ J 0 (o),y=y(o) \ dy J 



H = 

/?(0) ^y-y(O) 



f dHf3) 

\ df3 



) 3(0) 

(C.8) 



and the relation = (y — y^^^) — e we obtain the linear constraints 
w{(3, y) = w{(3^ ^ -I- AA/d + BAy or 



w(^,y) = Cy, + AAf3-Be 
h0) = Ch + H^ 



(C.9) 

(C.IO) 



with 

+ B{y -y^°^) and Ch = h0^°^) (C.ll) 



are the contradictions between the approximate values for the unknown pa- 
rameters and the given observations and among the approximate values for 
the unknowns. 

Setting the partials of <l> from (C.7) zero yields 

1 T T T 

— = -Qyy^e+ B^\ = 0 — ^ = A^X+ H^y = 0 (C.12) 

dy d'j3 

Q(J) . — . Q(p . — . 

— ^ = c^ + AA!3-Be = {) -^ = Ch + HAf3 = Q (C.13) 

dA ' og' 



192 C Statistics 



From (C.12a) follows the relation 

e=QyyB'^X (C.14) 

When substituting (C.14) into (C.13a), solving for A yields 

\={BQyyB^)-\c^ + A^) (C.15) 

Substitution in (C.12b) yields the symmetric normal equation system 



/ A^iBQyyB'^)-^A I -A'^iBQyyB^)-^ 

\ H o/Vm/V -Ch 

The Lagrangian multipliers can be obtained from (C.15) which then yields the 
estimated residuals e in (C.14). Using the residuals, the fitted observations 
are given by y = y — e. To impose the constraints g(y) = 0, we solve a 
reduced GauB-Helmert model with no unknowns, then w'(y,/3) = g(/3) and 
of course h'(/3) undefined. Then the new estimation y is 

y = y - i:yyG'^(Gi:yyG)'ig(y) 




The estimated variance factor is given by 



^2 

a = 



G + H-U 



(C.17) 



The number R of constraints above the number U — H , which is necessary 
for determining the unknown parameters, the redundancy is the denominator 
R = G — {U — H). We finally obtain the estimated covariance matrix 






/ 3/3 



(C.18) 



of the estimated parameters, where results from the inverted reduced 
normal equation matrix using N = AJ {BQyyB^)~^ A 



S T ) \H 0 



(C.19) 



We have summarized the approach in algorithm in table C.2. For a criterion to 

'(0 

stop the iteration we require that the changes Z\/3 to the estimation should 

' — (b 

be less than 1% with respect to their uncertainty, thus A(3 /trace( ^bb) < 
10-2. 
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Iterative Optimization using the Gaufi-Helmert model 

Objective: Compute an estimate for an unknown vector /3 and an observa- 
tion vector y using the Gauss-Helmert Model, see section C.2. 



1 W = dim(«;); H — dim(h); U = dim(/3) 

2 yO) — y j j given observations 

3/3 // initial value of unknowns 

4 for i from 0 to Maxiterl do 

5 // compute Jacobians of constraint 

6 I _ 

l(S«./3'’) 

7 B= 

sy l®w./3”) 
g _ Sh(l3) ' 



afi 1(3'^’) 



9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 






Cn, =Wj{y''"’,(3 ) + B{y-y^^>) 



Ch = h0 * ) 
^ww = (B S 
= (i) 



yy 



Aj3' 






0 



^ WW'^'W 

i 



— ^ww + AAf3 ) 
e = Syy B^ X 

yO+1) = y _ e 

for 1 from 0 to Maxlter2 do 

Q _ 9g(y ) I 



Sy 



;(*+!) 



I(S«) 



V =V 
done 



- JiO+i) 



(0, 



-i;yyG2GSyyG^)-ig(y'') 



17^ = Syy'e or e^S^A or 

23 = f2V(W^ + - t/) 

24 last if A(3^ ^ /trace (A/bb) < 10“^ 

25 done 

26 return (3' , y' 






Algorithm C.2: Iterative Solution for the optimization problem using the Gaufi- 
Helmert model, see text for details. 
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C.2.2 General Gauss Helmert Model with Block Structure 

We now want to introduce a variant of the above solution, which is given as 
follows: assume an unknown object f3 with n blocks of observations, where 
each block contains mj observation sub-vector y^j^k) of the same type, where 
j = I,--- ,n, fc = 1, • • • ,mj. The observation model function are 

given for each observation sub- vector, where the function remains identical 
within each block j. The observation model function is bilinear of the form 

= Aj{y^k)l3 = Bj(f3)yj^k 

The covariance matrices for each observation sub- vector Syy^Q^k)i that are 
uncorrelated among each other. 

With the above assumptions, the computational cost of the iterative estima- 
tion can be greatly reduced due to the special structures of the Jacobians A, 
B and the covariance matrix Syy. 



/\ = diag(/\i(yi_i),--- , i), • • • ,/li(y„,^J) 
e = diag(ei(/3),... ,ei(/3)) 

^yy ~ dtag * * * ? ‘ * * ^ yy^{n,m-n)i ' ' ' : ^ yy,{n,m.n)') 

(C.20) 

Then the upper left sub-matrix of the normal equation (C.16) can be written 
as the sum of components: 

A {BQyyB ) A= (^B(^jk) ^yy,(hk) B(^j^k)j 

(j,k) 

Additionally, one can efficiently compute the fitted observations y and obtains 
a significant speed-up, especially when the redundancy is large. The modified 
algorithm is listed in table C.3. 
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Iterative Block-Optimization using the Gaufi-Helmert model 

Objective: Compute an estimate for an unknown vector /3 and an observa- 
tion vector y using the Gauss-Helmert Model, see section C.2. This algo- 
rithm takes advantage of the existence of blocks of independent observa- 
tions, see text for details. 



1 W = dim(iOj); H = dim(h); U = dim(,3); G — dim(iOj) 

2 vfl = V(i,k) 1 1 given observations 

3 (^'*11 initial value of unknowns 

4 for i from 0 to Maxiterl do 

5 // compute Jacobians of constraint 

9wj(i/,/3) 

a/3 ,0.1 a(») 



15 

16 



19 



20 












avjj(y,(5) 



dy 



(0 



u _ 9h.(f3) 



(0 



sO.fe) = + B{y - 



5(0 X 



^ww,(j,k) ^yy,U,k) ^(j,k)) 



4i/3 \ _ I 12(j,k) ^U,k)-^ujw'^(j,k) 









•^(i + 1) -^(i) (i) 

13 l3 =/3 +Ai3 



H 






-Ch 





14 ^{j,k) = ^ium,(j,k) +'4(j,fc)/\/3 ) 



e ^■11 



^yy.U.k) ^{j,k)'^ 

y(j,k) y(j,k) ^(j,k) 



17 ^(0) 

17 V(j,k) = v \ j , k ) 

18 for 1 from 0 to Maxiter do 



r- 8gi(y) 

= L. 



dy 



y(j,k) —y( 3 \k) ^yy^U,k)i^U,k)^y(i,k)y(,i,k)^U,kv"> ^Sj{y(j,k)) 






(0 



21 done 

22 
23 



12= = 



(j,k) ^U,k) °(3,k)^U,k) 



n= = n^/{W + H-U) 

24 last if Af3^ ^ /trsice{Shb) < 10“= 

25 done 

26 return (3 “ 



Algorithm C.3: Iterative Solution for the optimization problem using the Gaufi- 
Helmert model. In contrast to the algorithm in table C.2, we assume blocks of 
independent observations, see text for details. 
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